[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:13:28 dist-test-slave-mkt3 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:13:28 dist-test-slave-mkt3 krb5kdc[395](info): set up 2 sockets
Jun 19 13:13:28 dist-test-slave-mkt3 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 13:13:29.440853 388 mini_kdc.cc:121] Time spent starting KDC: real 2.060s user 0.010s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 13:13:29.602154 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.161s user 0.001s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 13:13:29.773191 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.171s user 0.002s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260619 13:13:30.106786 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.333s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/test-user.keytab.
W20260619 13:13:30.652446 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.545s user 0.003s sys 0.000s
Jun 19 13:13:30 dist-test-slave-mkt3 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:13:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:13:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:13:30.715663 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35419
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:31.025288 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:31.025576 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:31.025691 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:31.032738 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:13:31.032846 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:31.032902 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:31.032979 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:13:31.033041 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:13:31.043605 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:31.044963 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:31.046288 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:31.058125 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:31.058153 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:31.058806 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:31.058848 408 server_base.cc:1034] running on GCE node
I20260619 13:13:31.059583 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:31.060894 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:31.062080 408 hybrid_clock.cc:648] HybridClock initialized: now 1781874811061985 us; error 98 us; skew 500 ppm
I20260619 13:13:31.062379 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:31 dist-test-slave-mkt3 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:31.065157 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:13:31.066591 408 webserver.cc:460] Webserver started at http://127.0.97.62:38827/ using document root <none> and password file <none>
I20260619 13:13:31.067274 408 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:31.067385 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:31.067652 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:31.070482 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "904b7cc3408446a7b501410445297b5e"
format_stamp: "Formatted at 2026-06-19 13:13:31 on dist-test-slave-mkt3"
server_key: "9d5d849b21f25af2539b36d0ebbaf373"
server_key_iv: "219ff86e8bc8c20969dad08133ecae8d"
server_key_version: "encryptionkey@0"
I20260619 13:13:31.071254 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "904b7cc3408446a7b501410445297b5e"
format_stamp: "Formatted at 2026-06-19 13:13:31 on dist-test-slave-mkt3"
server_key: "9d5d849b21f25af2539b36d0ebbaf373"
server_key_iv: "219ff86e8bc8c20969dad08133ecae8d"
server_key_version: "encryptionkey@0"
I20260619 13:13:31.076505 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:13:31.079779 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:31.081060 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:13:31.081208 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "904b7cc3408446a7b501410445297b5e"
format_stamp: "Formatted at 2026-06-19 13:13:31 on dist-test-slave-mkt3"
server_key: "9d5d849b21f25af2539b36d0ebbaf373"
server_key_iv: "219ff86e8bc8c20969dad08133ecae8d"
server_key_version: "encryptionkey@0"
I20260619 13:13:31.081379 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:31.100207 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:31.104314 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:31.104579 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:31.122454 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35419
I20260619 13:13:31.122473 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35419 every 8 connection(s)
I20260619 13:13:31.123775 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:13:31.127458 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:31.128851 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 408
I20260619 13:13:31.129172 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:13:31.131593 388 external_mini_cluster.cc:1428] Setting key b777aeb10bd870d879b11cfac190d959
I20260619 13:13:31.139992 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e: Bootstrap starting.
I20260619 13:13:31.142855 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:31.144009 475 log.cc:824] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:31.146984 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e: No bootstrap required, opened a new log
I20260619 13:13:31.155906 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904b7cc3408446a7b501410445297b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35419 } }
I20260619 13:13:31.156401 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:31.156517 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 904b7cc3408446a7b501410445297b5e, State: Initialized, Role: FOLLOWER
I20260619 13:13:31.157085 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904b7cc3408446a7b501410445297b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35419 } }
I20260619 13:13:31.157279 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:13:31.157390 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:13:31.157547 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:31.159718 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904b7cc3408446a7b501410445297b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35419 } }
I20260619 13:13:31.160209 475 leader_election.cc:302] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 904b7cc3408446a7b501410445297b5e; no voters:
Jun 19 13:13:31 dist-test-slave-mkt3 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:31.160815 475 leader_election.cc:288] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:13:31.160912 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:31.162060 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [term 1 LEADER]: Becoming Leader. State: Replica: 904b7cc3408446a7b501410445297b5e, State: Running, Role: LEADER
I20260619 13:13:31.162613 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904b7cc3408446a7b501410445297b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35419 } }
I20260619 13:13:31.162966 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:13:31.173113 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:31.136257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36456 (local address 127.0.97.62:35419)
0619 13:13:31.137122 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:31.137139 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:31.138365 (+ 1226us) server_negotiation.cc:407] Connection header received
0619 13:13:31.147580 (+ 9215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:31.147605 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:31.148050 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:31.148508 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:31.151109 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:31.153102 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:31.155150 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:31.156160 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:31.156437 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:31.162510 (+ 6073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:31.162521 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:31.162540 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:31.162593 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:31.167435 (+ 4842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:31.168499 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:31.168507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:31.168516 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:31.168760 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:31.169305 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:31.169310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:31.169314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:31.170057 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:31.170377 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:31.171377 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0619 13:13:31.171795 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":354,"threads_started":1}
I20260619 13:13:31.175660 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 904b7cc3408446a7b501410445297b5e. Latest consensus state: current_term: 1 leader_uuid: "904b7cc3408446a7b501410445297b5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904b7cc3408446a7b501410445297b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35419 } } }
I20260619 13:13:31.175697 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "904b7cc3408446a7b501410445297b5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904b7cc3408446a7b501410445297b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35419 } } }
I20260619 13:13:31.176441 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:31.176441 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:31.177685 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:13:31.182050 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:13:31.190802 494 catalog_manager.cc:1347] Generated new cluster ID: 69c939e72afd4119b5ca963aaadc480e
I20260619 13:13:31.190898 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:13:31.207326 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:13:31.208570 494 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:13:31.215070 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 904b7cc3408446a7b501410445297b5e: Generated new TSK 0
I20260619 13:13:31.215791 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:13:31.258527 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35419
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:31.566718 502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:31.566987 502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:31.567116 502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:31.573865 502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:31.574000 502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:31.574158 502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:13:31.583529 502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35419
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:31.584748 502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:31.586196 502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:31.598789 510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:31.598874 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:31.598856 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:31.599620 502 server_base.cc:1034] running on GCE node
I20260619 13:13:31.600520 502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:31.601374 502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:31.602553 502 hybrid_clock.cc:648] HybridClock initialized: now 1781874811602503 us; error 46 us; skew 500 ppm
I20260619 13:13:31.602821 502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:31 dist-test-slave-mkt3 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:31.605623 502 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:13:31.606961 502 webserver.cc:460] Webserver started at http://127.0.97.1:36587/ using document root <none> and password file <none>
I20260619 13:13:31.607601 502 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:31.607728 502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:31.608007 502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:31.610725 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "24ebd87623e14758a0134e2cfc2fb1cb"
format_stamp: "Formatted at 2026-06-19 13:13:31 on dist-test-slave-mkt3"
server_key: "36c70fddce47daf6a6fba813e483253c"
server_key_iv: "9c05149e97ef729ce78191dfb01e32de"
server_key_version: "encryptionkey@0"
I20260619 13:13:31.611438 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "24ebd87623e14758a0134e2cfc2fb1cb"
format_stamp: "Formatted at 2026-06-19 13:13:31 on dist-test-slave-mkt3"
server_key: "36c70fddce47daf6a6fba813e483253c"
server_key_iv: "9c05149e97ef729ce78191dfb01e32de"
server_key_version: "encryptionkey@0"
I20260619 13:13:31.616093 502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:13:31.619190 517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:31.620433 502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 13:13:31.620571 502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "24ebd87623e14758a0134e2cfc2fb1cb"
format_stamp: "Formatted at 2026-06-19 13:13:31 on dist-test-slave-mkt3"
server_key: "36c70fddce47daf6a6fba813e483253c"
server_key_iv: "9c05149e97ef729ce78191dfb01e32de"
server_key_version: "encryptionkey@0"
I20260619 13:13:31.620764 502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:31.642804 502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:31.646624 502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:31.646834 502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:31.647639 502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:13:31.648774 502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:31.648844 502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:31.648926 502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:31.648965 502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:31.679490 502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38775
I20260619 13:13:31.679535 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38775 every 8 connection(s)
I20260619 13:13:31.680708 502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:13:31.681866 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 502
I20260619 13:13:31.682076 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:13:31.682786 388 external_mini_cluster.cc:1428] Setting key 1ced25f7e46df0dc8cd18239cea90f16
Jun 19 13:13:31 dist-test-slave-mkt3 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:13:31.702041 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:31.683264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60729 (local address 127.0.97.62:35419)
0619 13:13:31.684320 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:31.684326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:31.684722 (+ 396us) server_negotiation.cc:407] Connection header received
0619 13:13:31.687178 (+ 2456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:31.687182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:31.687244 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:31.687494 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:31.689329 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:31.690062 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:31.691827 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:31.692434 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:31.692550 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:31.695851 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:31.695857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:31.695862 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:31.695901 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:31.698936 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:31.699924 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:31.699928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:31.699930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:31.700034 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:31.700367 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:31.700370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:31.700372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:31.700615 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:31.700767 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:31.701454 (+ 687us) server_negotiation.cc:299] Negotiation successful
0619 13:13:31.701655 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":775,"threads_started":1}
I20260619 13:13:31.702927 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:31.683835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35419 (local address 127.0.97.1:60729)
0619 13:13:31.684588 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:31.684631 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:31.686850 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:31.687763 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:31.687775 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:31.688421 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:31.689146 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:31.689170 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:31.690204 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:31.690210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:31.691682 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:31.691694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:31.692608 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:31.692612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:31.692802 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:31.693415 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:31.693435 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:31.695622 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:31.699041 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:31.699047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:31.699059 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:31.699832 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:31.700109 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:31.700112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:31.700115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:31.700290 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:31.700705 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:31.700710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:31.701089 (+ 379us) client_negotiation.cc:769] Sending connection context
0619 13:13:31.701355 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 13:13:31.701826 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":310,"threads_started":1}
I20260619 13:13:31.704396 630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35419
I20260619 13:13:31.704661 630 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:31.705365 630 heartbeater.cc:511] Master 127.0.97.62:35419 requested a full tablet report, sending...
I20260619 13:13:31.707386 440 ts_manager.cc:194] Registered new tserver with Master: 24ebd87623e14758a0134e2cfc2fb1cb (127.0.97.1:38775)
I20260619 13:13:31.708750 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:13:31.718559 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35419
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:32.027179 636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:32.027456 636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:32.027576 636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:32.034718 636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:32.034823 636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:32.034974 636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:13:32.044781 636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35419
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:32.045974 636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:32.047230 636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:32.059051 642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:32.059084 641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:32.059055 644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:32.060230 636 server_base.cc:1034] running on GCE node
I20260619 13:13:32.060801 636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:32.061720 636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:32.062873 636 hybrid_clock.cc:648] HybridClock initialized: now 1781874812062833 us; error 61 us; skew 500 ppm
I20260619 13:13:32.063161 636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:32 dist-test-slave-mkt3 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:32.066461 636 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:13:32.067807 636 webserver.cc:460] Webserver started at http://127.0.97.2:45795/ using document root <none> and password file <none>
I20260619 13:13:32.068444 636 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:32.068557 636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:32.068845 636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:32.071656 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "c287f10493884eac872424ea5b5ddcab"
format_stamp: "Formatted at 2026-06-19 13:13:32 on dist-test-slave-mkt3"
server_key: "4d28db3892ddebff0f8d9e88ea33caef"
server_key_iv: "b1802e97665afbba86562c534b18bd10"
server_key_version: "encryptionkey@0"
I20260619 13:13:32.072372 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c287f10493884eac872424ea5b5ddcab"
format_stamp: "Formatted at 2026-06-19 13:13:32 on dist-test-slave-mkt3"
server_key: "4d28db3892ddebff0f8d9e88ea33caef"
server_key_iv: "b1802e97665afbba86562c534b18bd10"
server_key_version: "encryptionkey@0"
I20260619 13:13:32.077306 636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 13:13:32.080830 651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:32.082190 636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:13:32.082336 636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "c287f10493884eac872424ea5b5ddcab"
format_stamp: "Formatted at 2026-06-19 13:13:32 on dist-test-slave-mkt3"
server_key: "4d28db3892ddebff0f8d9e88ea33caef"
server_key_iv: "b1802e97665afbba86562c534b18bd10"
server_key_version: "encryptionkey@0"
I20260619 13:13:32.082532 636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:32.100831 636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:32.104544 636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:32.104787 636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:32.105664 636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:13:32.106926 636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:32.106999 636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:32.107106 636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:32.107157 636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:32.140589 763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33911 every 8 connection(s)
I20260619 13:13:32.141263 636 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33911
I20260619 13:13:32.142587 636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:13:32.152032 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 636
I20260619 13:13:32.152222 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:13:32.152760 388 external_mini_cluster.cc:1428] Setting key 6702f112b8f7c1d525a7b4a2c019e0c5
Jun 19 13:13:32 dist-test-slave-mkt3 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:32.165196 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:32.145382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52087 (local address 127.0.97.62:35419)
0619 13:13:32.145559 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:32.145565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:32.146857 (+ 1292us) server_negotiation.cc:407] Connection header received
0619 13:13:32.149319 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:32.149323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:32.149391 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:32.149482 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:32.151847 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.152930 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.155027 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.155583 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.155699 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:32.159452 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:32.159456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:32.159459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:32.159498 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:32.162253 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:32.163020 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:32.163026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:32.163029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:32.163168 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:32.163587 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:32.163592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:32.163596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:32.163849 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:32.163969 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:32.164677 (+ 708us) server_negotiation.cc:299] Negotiation successful
0619 13:13:32.164886 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:13:32.166234 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:32.145930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35419 (local address 127.0.97.2:52087)
0619 13:13:32.146668 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:32.146725 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:32.149018 (+ 2293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:32.149729 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:32.149741 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:32.150899 (+ 1158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:32.151634 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:32.151681 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.153106 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.153110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:32.154891 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:32.154903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.155807 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.155813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:32.156029 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:32.156669 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:32.156696 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:32.159212 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:32.162398 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:32.162424 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:32.162443 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:32.162878 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:32.163289 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:32.163294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:32.163298 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:32.163475 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:32.163967 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:32.163975 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:32.164343 (+ 368us) client_negotiation.cc:769] Sending connection context
0619 13:13:32.164575 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 13:13:32.165094 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":253,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:13:32.167659 764 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35419
I20260619 13:13:32.167948 764 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:32.168659 764 heartbeater.cc:511] Master 127.0.97.62:35419 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 13:13:32.170017 440 ts_manager.cc:194] Registered new tserver with Master: c287f10493884eac872424ea5b5ddcab (127.0.97.2:33911)
I20260619 13:13:32.170789 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:13:32.187088 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35419
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:32.494169 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:32.494423 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:32.494550 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:32.501417 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:32.501529 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:32.501684 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:13:32.511487 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35419
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:32.512705 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:32.513975 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:32.526284 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:32.526382 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:32.526352 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:32.527109 769 server_base.cc:1034] running on GCE node
I20260619 13:13:32.527635 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:32.528462 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:32.529644 769 hybrid_clock.cc:648] HybridClock initialized: now 1781874812529603 us; error 40 us; skew 500 ppm
I20260619 13:13:32.529945 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:32 dist-test-slave-mkt3 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:32.532685 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:13:32.533973 769 webserver.cc:460] Webserver started at http://127.0.97.3:44745/ using document root <none> and password file <none>
I20260619 13:13:32.534576 769 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:32.534667 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:32.534909 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:32.537504 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "b12b638ce19c4423b49827387d299b2e"
format_stamp: "Formatted at 2026-06-19 13:13:32 on dist-test-slave-mkt3"
server_key: "f63dc045424e6401c0b972b92c110ba8"
server_key_iv: "e8379f9f95f71a3a8de2d26bbfc93aaa"
server_key_version: "encryptionkey@0"
I20260619 13:13:32.538241 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b12b638ce19c4423b49827387d299b2e"
format_stamp: "Formatted at 2026-06-19 13:13:32 on dist-test-slave-mkt3"
server_key: "f63dc045424e6401c0b972b92c110ba8"
server_key_iv: "e8379f9f95f71a3a8de2d26bbfc93aaa"
server_key_version: "encryptionkey@0"
I20260619 13:13:32.543501 769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:13:32.547160 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:32.548404 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:13:32.548553 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "b12b638ce19c4423b49827387d299b2e"
format_stamp: "Formatted at 2026-06-19 13:13:32 on dist-test-slave-mkt3"
server_key: "f63dc045424e6401c0b972b92c110ba8"
server_key_iv: "e8379f9f95f71a3a8de2d26bbfc93aaa"
server_key_version: "encryptionkey@0"
I20260619 13:13:32.548730 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:32.563361 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:32.567436 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:32.567683 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:32.568603 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:13:32.569867 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:32.569952 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:32.570070 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:32.570123 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:32.602378 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36005
I20260619 13:13:32.602417 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36005 every 8 connection(s)
I20260619 13:13:32.603579 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:13:32.611043 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 769
I20260619 13:13:32.611231 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:13:32.611726 388 external_mini_cluster.cc:1428] Setting key dc17ea6f68644e2bea935893063b2182
Jun 19 13:13:32 dist-test-slave-mkt3 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:32.622375 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:32.605906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50787 (local address 127.0.97.62:35419)
0619 13:13:32.606059 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:32.606064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:32.607207 (+ 1143us) server_negotiation.cc:407] Connection header received
0619 13:13:32.609010 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:32.609015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:32.609078 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:32.609180 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:32.610593 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.611649 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.613304 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.613803 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.613936 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:32.617488 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:32.617493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:32.617496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:32.617526 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:32.619706 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:32.620318 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:32.620323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:32.620325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:32.620430 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:32.620801 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:32.620806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:32.620809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:32.621072 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:32.621200 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:32.621817 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 13:13:32.622040 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 13:13:32.623231 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:32.606394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35419 (local address 127.0.97.3:50787)
0619 13:13:32.607057 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:32.607093 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:32.608724 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:32.609376 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:32.609384 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:32.609880 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:32.610437 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:32.610454 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.611783 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.611787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:32.613177 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:32.613189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.614019 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.614027 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:32.614258 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:32.614844 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:32.614865 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:32.617294 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:32.619823 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:32.619832 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:32.619848 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:32.620214 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:32.620522 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:32.620527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:32.620531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:32.620704 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:32.621184 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:32.621191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:32.621486 (+ 295us) client_negotiation.cc:769] Sending connection context
0619 13:13:32.621721 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 13:13:32.622219 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":252,"threads_started":1}
I20260619 13:13:32.624491 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35419
I20260619 13:13:32.624742 897 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:32.625423 897 heartbeater.cc:511] Master 127.0.97.62:35419 requested a full tablet report, sending...
I20260619 13:13:32.626838 440 ts_manager.cc:194] Registered new tserver with Master: b12b638ce19c4423b49827387d299b2e (127.0.97.3:36005)
I20260619 13:13:32.627409 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:13:32.627812 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50787
Jun 19 13:13:32 dist-test-slave-mkt3 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874812, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 13:13:32 dist-test-slave-mkt3 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874812, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:13:32.670257 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:32.652292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.0.97.1:38775)
0619 13:13:32.652933 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:32.652940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:32.652996 (+ 56us) server_negotiation.cc:407] Connection header received
0619 13:13:32.653107 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:32.653114 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:32.653350 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:32.653496 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:32.655298 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.656359 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.657499 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.658062 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.658153 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:32.665366 (+ 7213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:32.665376 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:32.665390 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:32.665428 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:32.667953 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:32.668535 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:32.668542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:32.668544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:32.668658 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:32.668955 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:32.668958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:32.668960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:32.669271 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:32.669476 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:32.669755 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 13:13:32.669973 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":402,"threads_started":1}
W20260619 13:13:32.671418 545 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34594
Jun 19 13:13:32 dist-test-slave-mkt3 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 13:13:32 dist-test-slave-mkt3 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:13:32.710956 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:32.698837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.0.97.1:38775)
0619 13:13:32.699052 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:32.699059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:32.699294 (+ 235us) server_negotiation.cc:407] Connection header received
0619 13:13:32.699436 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:32.699440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:32.699528 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:32.699612 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:32.700879 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.701546 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.702693 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.703070 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.703159 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:32.706674 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:32.706679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:32.706683 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:32.706713 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:32.708974 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:32.709527 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:32.709530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:32.709533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:32.709644 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:32.709935 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:32.709938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:32.709940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:32.710171 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:32.710326 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:32.710551 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 13:13:32.710721 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:13:32.711927 630 heartbeater.cc:503] Master 127.0.97.62:35419 was elected leader, sending a full tablet report...
I20260619 13:13:32.712205 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 502
I20260619 13:13:32.723798 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:32.713183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.0.97.1:38775)
0619 13:13:32.713348 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:32.713354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:32.713654 (+ 300us) server_negotiation.cc:407] Connection header received
0619 13:13:32.713764 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:32.713768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:32.713817 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:32.713922 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:32.714919 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.715561 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.716715 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.717120 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.717211 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:32.719541 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:32.719546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:32.719549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:32.719576 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:32.721900 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:32.722438 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:32.722441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:32.722443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:32.722549 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:32.722807 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:32.722810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:32.722811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:32.723025 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:32.723174 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:32.723376 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 13:13:32.723514 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:13:32.724438 625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:13:32.805332 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 502
I20260619 13:13:32.818230 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 636
Jun 19 13:13:32 dist-test-slave-mkt3 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:13:32.832705 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:32.819096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32936 (local address 127.0.97.2:33911)
0619 13:13:32.819615 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:32.819623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:32.819650 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:13:32.819705 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:32.819711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:32.819906 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:32.820030 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:32.821070 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.822154 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.823376 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.823918 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.824011 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:32.827762 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:32.827773 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:32.827787 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:32.827826 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:32.830513 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:32.831083 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:32.831090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:32.831092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:32.831202 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:32.831465 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:32.831468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:32.831470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:32.831778 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:32.831984 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:32.832173 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 13:13:32.832357 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":270,"threads_started":1}
I20260619 13:13:32.834040 759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:13:32.913551 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 636
I20260619 13:13:32.931439 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 769
Jun 19 13:13:32 dist-test-slave-mkt3 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:13:32.946151 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:32.932324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.0.97.3:36005)
0619 13:13:32.932887 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:32.932895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:32.932922 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:13:32.932977 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:32.932983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:32.933183 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:32.933326 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:32.934532 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.935582 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.936808 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:32.937366 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:32.937458 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:32.941103 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:32.941113 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:32.941128 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:32.941159 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:32.943719 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:32.944280 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:32.944287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:32.944290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:32.944406 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:32.944705 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:32.944709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:32.944712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:32.945027 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:32.945247 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:32.945444 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 13:13:32.945623 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":285,"threads_started":1}
I20260619 13:13:32.947415 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:13:33.032547 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 769
I20260619 13:13:33.045183 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 408
I20260619 13:13:33.046284 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:13:33.144063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 408
2026-06-19T13:13:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5808 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:13:33 dist-test-slave-mkt3 krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:13:33 dist-test-slave-mkt3 krb5kdc[918](info): set up 2 sockets
Jun 19 13:13:33 dist-test-slave-mkt3 krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260619 13:13:34.121544 388 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.003s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:13:34 dist-test-slave-mkt3 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:13:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:13:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:13:34.274966 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37513
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:34.576905 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:34.577193 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:34.577306 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:34.584411 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:13:34.584519 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:34.584571 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:34.584648 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:13:34.584707 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:13:34.594291 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:34.595575 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:34.596855 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:34.607548 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:34.607487 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:34.607509 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:34.608306 931 server_base.cc:1034] running on GCE node
I20260619 13:13:34.609047 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:34.610430 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:34.611629 931 hybrid_clock.cc:648] HybridClock initialized: now 1781874814611577 us; error 39 us; skew 500 ppm
I20260619 13:13:34.611953 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:34 dist-test-slave-mkt3 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:34.614781 931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:13:34.616286 931 webserver.cc:460] Webserver started at http://127.0.97.62:34121/ using document root <none> and password file <none>
I20260619 13:13:34.616951 931 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:34.617062 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:34.617352 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:34.620281 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "0a4a3fb410a74f438838f3afcb986cae"
format_stamp: "Formatted at 2026-06-19 13:13:34 on dist-test-slave-mkt3"
server_key: "e2ed8a5895eaa26b63c7ccf6aa17e450"
server_key_iv: "5b33bc96eb00a823d2016ccb9cb25a63"
server_key_version: "encryptionkey@0"
I20260619 13:13:34.621019 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "0a4a3fb410a74f438838f3afcb986cae"
format_stamp: "Formatted at 2026-06-19 13:13:34 on dist-test-slave-mkt3"
server_key: "e2ed8a5895eaa26b63c7ccf6aa17e450"
server_key_iv: "5b33bc96eb00a823d2016ccb9cb25a63"
server_key_version: "encryptionkey@0"
I20260619 13:13:34.626010 931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260619 13:13:34.629423 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:34.630802 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:13:34.630957 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "0a4a3fb410a74f438838f3afcb986cae"
format_stamp: "Formatted at 2026-06-19 13:13:34 on dist-test-slave-mkt3"
server_key: "e2ed8a5895eaa26b63c7ccf6aa17e450"
server_key_iv: "5b33bc96eb00a823d2016ccb9cb25a63"
server_key_version: "encryptionkey@0"
I20260619 13:13:34.631155 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:34.654496 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:34.658207 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:34.658447 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:34.674728 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37513
I20260619 13:13:34.674749 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37513 every 8 connection(s)
I20260619 13:13:34.676059 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:13:34.679550 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 931
I20260619 13:13:34.679740 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:34.679790 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:13:34.680547 388 external_mini_cluster.cc:1428] Setting key c8c7a072bfc0884149ede6dc803dce7a
I20260619 13:13:34.692771 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae: Bootstrap starting.
Jun 19 13:13:34 dist-test-slave-mkt3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:34.696203 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:34.697467 998 log.cc:824] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:34.700548 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae: No bootstrap required, opened a new log
I20260619 13:13:34.703972 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:34.682461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59482 (local address 127.0.97.62:37513)
0619 13:13:34.683205 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:34.683217 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:34.683255 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:13:34.685450 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:34.685472 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:34.685920 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:34.686271 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:34.687889 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:34.689163 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:34.690881 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:34.691540 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:34.691674 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:34.695966 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:34.695977 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:34.695996 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:34.696042 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:34.699541 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:34.700194 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:34.700202 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:34.700210 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:34.700400 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:34.700818 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:34.700822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:34.700825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:34.701498 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:34.701795 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:34.702231 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 13:13:34.702661 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":321,"threads_started":1}
I20260619 13:13:34.709264 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4a3fb410a74f438838f3afcb986cae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37513 } }
I20260619 13:13:34.709576 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:34.709643 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4a3fb410a74f438838f3afcb986cae, State: Initialized, Role: FOLLOWER
I20260619 13:13:34.710170 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4a3fb410a74f438838f3afcb986cae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37513 } }
I20260619 13:13:34.710376 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:13:34.710489 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:13:34.710666 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:34.712756 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4a3fb410a74f438838f3afcb986cae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37513 } }
I20260619 13:13:34.713179 998 leader_election.cc:302] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a4a3fb410a74f438838f3afcb986cae; no voters:
I20260619 13:13:34.713680 998 leader_election.cc:288] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:13:34.713807 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:34.714421 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [term 1 LEADER]: Becoming Leader. State: Replica: 0a4a3fb410a74f438838f3afcb986cae, State: Running, Role: LEADER
I20260619 13:13:34.714934 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4a3fb410a74f438838f3afcb986cae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37513 } }
I20260619 13:13:34.715521 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:13:34.717564 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a4a3fb410a74f438838f3afcb986cae. Latest consensus state: current_term: 1 leader_uuid: "0a4a3fb410a74f438838f3afcb986cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4a3fb410a74f438838f3afcb986cae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37513 } } }
I20260619 13:13:34.717923 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:34.718797 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a4a3fb410a74f438838f3afcb986cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4a3fb410a74f438838f3afcb986cae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37513 } } }
I20260619 13:13:34.719167 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:34.722844 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:13:34.728171 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:13:34.735920 1010 catalog_manager.cc:1347] Generated new cluster ID: c1ee686aaa934f68a12cfa0c941ee3f0
I20260619 13:13:34.736032 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:13:34.762583 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:13:34.764457 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:13:34.775135 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a4a3fb410a74f438838f3afcb986cae: Generated new TSK 0
I20260619 13:13:34.776182 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:13:34.816928 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37513
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:35.131865 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:35.132138 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:35.132230 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:35.139397 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:35.139506 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:35.139634 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:13:35.149052 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37513
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:35.150278 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:35.151665 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:35.164158 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:35.164197 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:35.164119 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:35.165071 1024 server_base.cc:1034] running on GCE node
I20260619 13:13:35.165694 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:35.166664 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:35.167881 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781874815167827 us; error 45 us; skew 500 ppm
I20260619 13:13:35.168191 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:35 dist-test-slave-mkt3 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:35.171231 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:13:35.172605 1024 webserver.cc:460] Webserver started at http://127.0.97.1:46679/ using document root <none> and password file <none>
I20260619 13:13:35.173250 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:35.173350 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:35.173606 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:35.176512 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "4e311c022bd34908b4dd33e44d9abbba"
format_stamp: "Formatted at 2026-06-19 13:13:35 on dist-test-slave-mkt3"
server_key: "dda04aaccdbc2dd1cc863f709256ed16"
server_key_iv: "e634a000fcf8c5693a7a6529d751cfba"
server_key_version: "encryptionkey@0"
I20260619 13:13:35.177273 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4e311c022bd34908b4dd33e44d9abbba"
format_stamp: "Formatted at 2026-06-19 13:13:35 on dist-test-slave-mkt3"
server_key: "dda04aaccdbc2dd1cc863f709256ed16"
server_key_iv: "e634a000fcf8c5693a7a6529d751cfba"
server_key_version: "encryptionkey@0"
I20260619 13:13:35.182782 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 13:13:35.186244 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:35.187603 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:13:35.187744 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "4e311c022bd34908b4dd33e44d9abbba"
format_stamp: "Formatted at 2026-06-19 13:13:35 on dist-test-slave-mkt3"
server_key: "dda04aaccdbc2dd1cc863f709256ed16"
server_key_iv: "e634a000fcf8c5693a7a6529d751cfba"
server_key_version: "encryptionkey@0"
I20260619 13:13:35.187932 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:35.202669 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:35.206777 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:35.207023 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:35.207907 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:13:35.209182 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:35.209259 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:35.209362 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:35.209416 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:35.238888 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36311
I20260619 13:13:35.238924 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36311 every 8 connection(s)
I20260619 13:13:35.240144 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:13:35.242339 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 1024
I20260619 13:13:35.242542 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:13:35.243185 388 external_mini_cluster.cc:1428] Setting key f78a6086e79607fbe6ac155ab87cc73c
Jun 19 13:13:35 dist-test-slave-mkt3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:13:35.260552 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:35.242574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46761 (local address 127.0.97.62:37513)
0619 13:13:35.242990 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:35.242997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:35.244079 (+ 1082us) server_negotiation.cc:407] Connection header received
0619 13:13:35.245882 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:35.245888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:35.245962 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:35.246072 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:35.247476 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:35.248421 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:35.250607 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:35.251003 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:35.251094 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:35.254807 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:35.254815 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:35.254820 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:35.254862 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:35.257884 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:35.258540 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:35.258546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:35.258550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:35.258695 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:35.259084 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:35.259088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:35.259090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:35.259385 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:35.259501 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:35.260143 (+ 642us) server_negotiation.cc:299] Negotiation successful
0619 13:13:35.260308 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":187,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:13:35.261703 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:35.243179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37513 (local address 127.0.97.1:46761)
0619 13:13:35.243932 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:35.243984 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:35.245601 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:35.246276 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:35.246283 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:35.246769 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:35.247329 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:35.247345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:35.248575 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:35.248580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:35.250482 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:35.250496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:35.251210 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:35.251217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:35.251481 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:35.252152 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:35.252178 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:35.254585 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:35.258017 (+ 3432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:35.258024 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:35.258037 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:35.258427 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:35.258791 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:35.258794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:35.258797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:35.258964 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:35.259490 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:35.259495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:35.259827 (+ 332us) client_negotiation.cc:769] Sending connection context
0619 13:13:35.260040 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 13:13:35.260493 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":300,"threads_started":1}
I20260619 13:13:35.263046 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37513
I20260619 13:13:35.263325 1152 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:35.264040 1152 heartbeater.cc:511] Master 127.0.97.62:37513 requested a full tablet report, sending...
I20260619 13:13:35.266002 963 ts_manager.cc:194] Registered new tserver with Master: 4e311c022bd34908b4dd33e44d9abbba (127.0.97.1:36311)
I20260619 13:13:35.267364 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:13:35.278834 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37513
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:35.584753 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:35.585021 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:35.585155 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:35.592238 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:35.592350 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:35.592501 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:13:35.602332 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37513
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:35.603749 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:35.605113 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:35.617075 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:35.617121 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:35.617152 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:35.618031 1158 server_base.cc:1034] running on GCE node
I20260619 13:13:35.618590 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:35.619488 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:35.620684 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781874815620642 us; error 30 us; skew 500 ppm
I20260619 13:13:35.620967 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:35 dist-test-slave-mkt3 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:35.623992 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:13:35.625384 1158 webserver.cc:460] Webserver started at http://127.0.97.2:38237/ using document root <none> and password file <none>
I20260619 13:13:35.626092 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:35.626205 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:35.626492 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:35.629247 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "2ecd65726659457d81db587f75d3d67f"
format_stamp: "Formatted at 2026-06-19 13:13:35 on dist-test-slave-mkt3"
server_key: "94d87e3fd6d2d178e9618db350e25348"
server_key_iv: "ba776533583b6eddae598ce269c5d36e"
server_key_version: "encryptionkey@0"
I20260619 13:13:35.630023 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2ecd65726659457d81db587f75d3d67f"
format_stamp: "Formatted at 2026-06-19 13:13:35 on dist-test-slave-mkt3"
server_key: "94d87e3fd6d2d178e9618db350e25348"
server_key_iv: "ba776533583b6eddae598ce269c5d36e"
server_key_version: "encryptionkey@0"
I20260619 13:13:35.634865 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:13:35.638103 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:35.639321 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 13:13:35.639461 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "2ecd65726659457d81db587f75d3d67f"
format_stamp: "Formatted at 2026-06-19 13:13:35 on dist-test-slave-mkt3"
server_key: "94d87e3fd6d2d178e9618db350e25348"
server_key_iv: "ba776533583b6eddae598ce269c5d36e"
server_key_version: "encryptionkey@0"
I20260619 13:13:35.639640 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:35.654824 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:35.658386 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:35.658638 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:35.659495 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:13:35.660714 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:35.660789 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:35.660903 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:35.660960 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:35.690651 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34723
I20260619 13:13:35.690667 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34723 every 8 connection(s)
I20260619 13:13:35.691965 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:13:35.693598 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 1158
I20260619 13:13:35.693784 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:13:35.694239 388 external_mini_cluster.cc:1428] Setting key bef25415fcf8fb52c34ba7997ac87962
Jun 19 13:13:35 dist-test-slave-mkt3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 13:13:35.713235 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:35.694554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48605 (local address 127.0.97.62:37513)
0619 13:13:35.695420 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:35.695424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:35.695990 (+ 566us) server_negotiation.cc:407] Connection header received
0619 13:13:35.698418 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:35.698422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:35.698485 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:35.698562 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:35.700370 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:35.701236 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:35.702759 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:35.703366 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:35.703481 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:35.706996 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:35.707003 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:35.707008 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:35.707048 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:35.710228 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:35.710956 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:35.710959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:35.710962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:35.711123 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:35.711542 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:35.711547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:35.711549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:35.711829 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:35.711977 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:35.712700 (+ 723us) server_negotiation.cc:299] Negotiation successful
0619 13:13:35.712920 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765}
I20260619 13:13:35.714263 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:35.695056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37513 (local address 127.0.97.2:48605)
0619 13:13:35.695824 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:35.695876 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:35.698141 (+ 2265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:35.698807 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:35.698818 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:35.699495 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:35.700205 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:35.700239 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:35.701388 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:35.701391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:35.702627 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:35.702638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:35.703539 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:35.703544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:35.703727 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:35.704325 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:35.704346 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:35.706766 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:35.710381 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:35.710391 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:35.710408 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:35.710843 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:35.711240 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:35.711244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:35.711247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:35.711431 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:35.711968 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:35.711976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:35.712338 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 13:13:35.712603 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 13:13:35.713105 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":283,"threads_started":1}
I20260619 13:13:35.715658 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37513
I20260619 13:13:35.715946 1286 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:35.716701 1286 heartbeater.cc:511] Master 127.0.97.62:37513 requested a full tablet report, sending...
I20260619 13:13:35.718072 963 ts_manager.cc:194] Registered new tserver with Master: 2ecd65726659457d81db587f75d3d67f (127.0.97.2:34723)
I20260619 13:13:35.718837 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:13:35.730971 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37513
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:36.037518 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:36.037802 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:36.037943 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:36.045125 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:36.045225 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:36.045372 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:13:36.057111 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37513
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:36.058454 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:36.059819 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:36.072420 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:36.072397 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:36.072540 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:36.073179 1291 server_base.cc:1034] running on GCE node
I20260619 13:13:36.073776 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:36.074759 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:36.076014 1291 hybrid_clock.cc:648] HybridClock initialized: now 1781874816075938 us; error 47 us; skew 500 ppm
I20260619 13:13:36.076378 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:36 dist-test-slave-mkt3 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:36.079429 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:13:36.080911 1291 webserver.cc:460] Webserver started at http://127.0.97.3:39761/ using document root <none> and password file <none>
I20260619 13:13:36.081633 1291 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:36.081740 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:36.082070 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:36.084883 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "83c63ec9c4a04ff78ee66b17f83411cc"
format_stamp: "Formatted at 2026-06-19 13:13:36 on dist-test-slave-mkt3"
server_key: "ad3cc8ea6aff77716f126a264d161f15"
server_key_iv: "7b654c950b0c59034c0ce30d08e40a75"
server_key_version: "encryptionkey@0"
I20260619 13:13:36.085642 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "83c63ec9c4a04ff78ee66b17f83411cc"
format_stamp: "Formatted at 2026-06-19 13:13:36 on dist-test-slave-mkt3"
server_key: "ad3cc8ea6aff77716f126a264d161f15"
server_key_iv: "7b654c950b0c59034c0ce30d08e40a75"
server_key_version: "encryptionkey@0"
I20260619 13:13:36.090524 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:13:36.093819 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:36.095167 1291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:13:36.095312 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "83c63ec9c4a04ff78ee66b17f83411cc"
format_stamp: "Formatted at 2026-06-19 13:13:36 on dist-test-slave-mkt3"
server_key: "ad3cc8ea6aff77716f126a264d161f15"
server_key_iv: "7b654c950b0c59034c0ce30d08e40a75"
server_key_version: "encryptionkey@0"
I20260619 13:13:36.095485 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:36.110965 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:36.115126 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:36.115411 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:36.116338 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:13:36.117614 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:36.117684 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:36.117796 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:36.117880 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 13:13:36.146405 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45817
I20260619 13:13:36.146438 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45817 every 8 connection(s)
I20260619 13:13:36.147614 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:13:36.156258 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 1291
I20260619 13:13:36.156451 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:13:36.156965 388 external_mini_cluster.cc:1428] Setting key 8716e2c040d55d5b4538400c673c353f
Jun 19 13:13:36 dist-test-slave-mkt3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:36.167397 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.149865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33533 (local address 127.0.97.62:37513)
0619 13:13:36.150016 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:36.150020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:36.151525 (+ 1505us) server_negotiation.cc:407] Connection header received
0619 13:13:36.153243 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:36.153248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:36.153322 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:36.153409 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:36.155012 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.155722 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.157832 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.158424 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.158521 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.162068 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:36.162074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:36.162077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:36.162108 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:36.164569 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.165227 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.165232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.165236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.165376 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.165775 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.165780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.165784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.166076 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:36.166206 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:36.166868 (+ 662us) server_negotiation.cc:299] Negotiation successful
0619 13:13:36.167068 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:13:36.168262 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.150338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37513 (local address 127.0.97.3:33533)
0619 13:13:36.151371 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:36.151412 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:36.152976 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:36.153638 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:36.153646 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:36.154279 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:36.154844 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:36.154862 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.155875 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.155880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:36.157661 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:36.157675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.158676 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.158682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:36.158948 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.159680 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:36.159710 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:36.161830 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:36.164693 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:36.164701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:36.164718 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:36.165094 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:36.165473 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:36.165478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:36.165482 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:36.165661 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:36.166194 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:36.166201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:36.166513 (+ 312us) client_negotiation.cc:769] Sending connection context
0619 13:13:36.166745 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 13:13:36.167219 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":642,"threads_started":1}
I20260619 13:13:36.169480 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37513
I20260619 13:13:36.169734 1419 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:36.170477 1419 heartbeater.cc:511] Master 127.0.97.62:37513 requested a full tablet report, sending...
I20260619 13:13:36.171988 962 ts_manager.cc:194] Registered new tserver with Master: 83c63ec9c4a04ff78ee66b17f83411cc (127.0.97.3:45817)
I20260619 13:13:36.172765 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33533
I20260619 13:13:36.183743 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:13:36.200969 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.189961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.0.97.62:37513)
0619 13:13:36.190113 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:36.190118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:36.190539 (+ 421us) server_negotiation.cc:407] Connection header received
0619 13:13:36.190664 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:36.190667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:36.190723 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:36.190805 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:36.192143 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.192943 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.194139 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.194549 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.194638 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.196592 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:36.196597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:36.196600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:36.196630 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:36.198903 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.199466 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.199469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.199472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.199603 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.199897 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.199900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.199902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.200172 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:36.200338 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:36.200590 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 13:13:36.200735 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:13:36.207424 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:13:36.213958 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:13:36.235764 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.225493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45817 (local address 127.0.0.1:48904)
0619 13:13:36.226800 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:36.226846 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:36.227131 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:36.228110 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:36.228115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:36.228145 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:36.228413 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:36.228421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.230241 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.230246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:36.232892 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:36.232904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.234912 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.234922 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:36.235117 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.235169 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 13:13:36.235300 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 13:13:36.235459 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"thread_start_us":1413,"threads_started":1}
I20260619 13:13:36.236465 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.225662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.0.97.3:45817)
0619 13:13:36.226779 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:36.226789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:36.226964 (+ 175us) server_negotiation.cc:407] Connection header received
0619 13:13:36.227365 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:36.227373 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:36.227623 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:36.227801 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:36.228564 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.230088 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.233047 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.234538 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.234701 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.234940 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:36.236011 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0619 13:13:36.236184 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":368,"threads_started":1}
I20260619 13:13:36.237623 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.225630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36311 (local address 127.0.0.1:60508)
0619 13:13:36.227182 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:36.227198 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:36.228992 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:36.229764 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:36.229769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:36.229792 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:36.231202 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:36.231225 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.232756 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.232760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:36.234917 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:36.234929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.236913 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.236920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:36.237073 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.237105 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 13:13:36.237188 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 13:13:36.237327 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1420,"thread_start_us":919,"threads_started":1}
I20260619 13:13:36.237668 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.226153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.0.97.1:36311)
0619 13:13:36.228895 (+ 2742us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:36.228908 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:36.228938 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:13:36.229176 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:36.229184 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:36.229405 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:36.229573 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:36.231378 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.232625 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.235068 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.236492 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.236617 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.236798 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:36.237286 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 13:13:36.237443 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2593,"thread_start_us":2462,"threads_started":1}
I20260619 13:13:36.239756 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.227438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34723 (local address 127.0.0.1:45378)
0619 13:13:36.231062 (+ 3624us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:36.231076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:36.231208 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:36.231794 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:36.231799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:36.231823 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:36.232235 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:36.232247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.233921 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.233927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:36.236617 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:36.236631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.239010 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.239016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:36.239174 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.239204 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 13:13:36.239290 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 13:13:36.239413 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3520,"mutex_wait_us":1,"thread_start_us":3370,"threads_started":1}
I20260619 13:13:36.240398 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.227628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45378 (local address 127.0.97.2:34723)
0619 13:13:36.230704 (+ 3076us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:36.230712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:36.231176 (+ 464us) server_negotiation.cc:407] Connection header received
0619 13:13:36.231372 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:36.231379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:36.231605 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:36.231804 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:36.232408 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.233724 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.236793 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.238822 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.238958 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.239128 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:36.239952 (+ 824us) server_negotiation.cc:299] Negotiation successful
0619 13:13:36.240129 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2927,"thread_start_us":2824,"threads_started":1}
I20260619 13:13:36.240504 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 2d06dbe1b65243c1ab327d14057e078d (DEFAULT_TABLE table=test-table [id=d99d1ac94e104ea3b77e6520355ec8bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:36.242136 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d06dbe1b65243c1ab327d14057e078d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:36.241981 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 2d06dbe1b65243c1ab327d14057e078d (DEFAULT_TABLE table=test-table [id=d99d1ac94e104ea3b77e6520355ec8bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:36.243368 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d06dbe1b65243c1ab327d14057e078d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:36.246871 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 2d06dbe1b65243c1ab327d14057e078d (DEFAULT_TABLE table=test-table [id=d99d1ac94e104ea3b77e6520355ec8bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:36.248111 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d06dbe1b65243c1ab327d14057e078d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:36.258042 1438 tablet_bootstrap.cc:492] T 2d06dbe1b65243c1ab327d14057e078d P 2ecd65726659457d81db587f75d3d67f: Bootstrap starting.
I20260619 13:13:36.260824 1438 tablet_bootstrap.cc:654] T 2d06dbe1b65243c1ab327d14057e078d P 2ecd65726659457d81db587f75d3d67f: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:36.261783 1438 log.cc:824] T 2d06dbe1b65243c1ab327d14057e078d P 2ecd65726659457d81db587f75d3d67f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:36.262594 1437 tablet_bootstrap.cc:492] T 2d06dbe1b65243c1ab327d14057e078d P 83c63ec9c4a04ff78ee66b17f83411cc: Bootstrap starting.
I20260619 13:13:36.262606 1439 tablet_bootstrap.cc:492] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba: Bootstrap starting.
I20260619 13:13:36.266333 1438 tablet_bootstrap.cc:492] T 2d06dbe1b65243c1ab327d14057e078d P 2ecd65726659457d81db587f75d3d67f: No bootstrap required, opened a new log
I20260619 13:13:36.266585 1438 ts_tablet_manager.cc:1397] T 2d06dbe1b65243c1ab327d14057e078d P 2ecd65726659457d81db587f75d3d67f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 13:13:36.266932 1437 tablet_bootstrap.cc:654] T 2d06dbe1b65243c1ab327d14057e078d P 83c63ec9c4a04ff78ee66b17f83411cc: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:36.267176 1439 tablet_bootstrap.cc:654] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:36.268090 1437 log.cc:824] T 2d06dbe1b65243c1ab327d14057e078d P 83c63ec9c4a04ff78ee66b17f83411cc: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:36.268193 1439 log.cc:824] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:36.269964 1152 heartbeater.cc:503] Master 127.0.97.62:37513 was elected leader, sending a full tablet report...
I20260619 13:13:36.271430 1439 tablet_bootstrap.cc:492] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba: No bootstrap required, opened a new log
I20260619 13:13:36.271441 1437 tablet_bootstrap.cc:492] T 2d06dbe1b65243c1ab327d14057e078d P 83c63ec9c4a04ff78ee66b17f83411cc: No bootstrap required, opened a new log
I20260619 13:13:36.271670 1437 ts_tablet_manager.cc:1397] T 2d06dbe1b65243c1ab327d14057e078d P 83c63ec9c4a04ff78ee66b17f83411cc: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260619 13:13:36.271795 1439 ts_tablet_manager.cc:1397] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 13:13:36.275697 1438 raft_consensus.cc:348] T 2d06dbe1b65243c1ab327d14057e078d P 2ecd65726659457d81db587f75d3d67f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c63ec9c4a04ff78ee66b17f83411cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45817 } } peers { permanent_uuid: "4e311c022bd34908b4dd33e44d9abbba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36311 } } peers { permanent_uuid: "2ecd65726659457d81db587f75d3d67f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34723 } }
I20260619 13:13:36.276258 1438 raft_consensus.cc:374] T 2d06dbe1b65243c1ab327d14057e078d P 2ecd65726659457d81db587f75d3d67f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:36.276553 1438 raft_consensus.cc:729] T 2d06dbe1b65243c1ab327d14057e078d P 2ecd65726659457d81db587f75d3d67f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecd65726659457d81db587f75d3d67f, State: Initialized, Role: FOLLOWER
I20260619 13:13:36.277191 1438 consensus_queue.cc:260] T 2d06dbe1b65243c1ab327d14057e078d P 2ecd65726659457d81db587f75d3d67f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c63ec9c4a04ff78ee66b17f83411cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45817 } } peers { permanent_uuid: "4e311c022bd34908b4dd33e44d9abbba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36311 } } peers { permanent_uuid: "2ecd65726659457d81db587f75d3d67f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34723 } }
I20260619 13:13:36.278280 1286 heartbeater.cc:503] Master 127.0.97.62:37513 was elected leader, sending a full tablet report...
I20260619 13:13:36.278966 1438 ts_tablet_manager.cc:1428] T 2d06dbe1b65243c1ab327d14057e078d P 2ecd65726659457d81db587f75d3d67f: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260619 13:13:36.280930 1437 raft_consensus.cc:348] T 2d06dbe1b65243c1ab327d14057e078d P 83c63ec9c4a04ff78ee66b17f83411cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c63ec9c4a04ff78ee66b17f83411cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45817 } } peers { permanent_uuid: "4e311c022bd34908b4dd33e44d9abbba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36311 } } peers { permanent_uuid: "2ecd65726659457d81db587f75d3d67f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34723 } }
I20260619 13:13:36.281378 1437 raft_consensus.cc:374] T 2d06dbe1b65243c1ab327d14057e078d P 83c63ec9c4a04ff78ee66b17f83411cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:36.281529 1437 raft_consensus.cc:729] T 2d06dbe1b65243c1ab327d14057e078d P 83c63ec9c4a04ff78ee66b17f83411cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c63ec9c4a04ff78ee66b17f83411cc, State: Initialized, Role: FOLLOWER
I20260619 13:13:36.281353 1439 raft_consensus.cc:348] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c63ec9c4a04ff78ee66b17f83411cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45817 } } peers { permanent_uuid: "4e311c022bd34908b4dd33e44d9abbba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36311 } } peers { permanent_uuid: "2ecd65726659457d81db587f75d3d67f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34723 } }
I20260619 13:13:36.281759 1439 raft_consensus.cc:374] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:36.281919 1439 raft_consensus.cc:729] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e311c022bd34908b4dd33e44d9abbba, State: Initialized, Role: FOLLOWER
I20260619 13:13:36.282133 1437 consensus_queue.cc:260] T 2d06dbe1b65243c1ab327d14057e078d P 83c63ec9c4a04ff78ee66b17f83411cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c63ec9c4a04ff78ee66b17f83411cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45817 } } peers { permanent_uuid: "4e311c022bd34908b4dd33e44d9abbba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36311 } } peers { permanent_uuid: "2ecd65726659457d81db587f75d3d67f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34723 } }
I20260619 13:13:36.282414 1439 consensus_queue.cc:260] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c63ec9c4a04ff78ee66b17f83411cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45817 } } peers { permanent_uuid: "4e311c022bd34908b4dd33e44d9abbba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36311 } } peers { permanent_uuid: "2ecd65726659457d81db587f75d3d67f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34723 } }
I20260619 13:13:36.283211 1419 heartbeater.cc:503] Master 127.0.97.62:37513 was elected leader, sending a full tablet report...
I20260619 13:13:36.283648 1437 ts_tablet_manager.cc:1428] T 2d06dbe1b65243c1ab327d14057e078d P 83c63ec9c4a04ff78ee66b17f83411cc: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260619 13:13:36.287066 1439 ts_tablet_manager.cc:1428] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260619 13:13:36.366467 1445 raft_consensus.cc:482] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:13:36.366750 1445 raft_consensus.cc:504] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c63ec9c4a04ff78ee66b17f83411cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45817 } } peers { permanent_uuid: "4e311c022bd34908b4dd33e44d9abbba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36311 } } peers { permanent_uuid: "2ecd65726659457d81db587f75d3d67f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34723 } }
I20260619 13:13:36.368391 1445 leader_election.cc:288] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c63ec9c4a04ff78ee66b17f83411cc (127.0.97.3:45817), 2ecd65726659457d81db587f75d3d67f (127.0.97.2:34723)
I20260619 13:13:36.374747 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.368853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36261 (local address 127.0.97.2:34723)
0619 13:13:36.369068 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:36.369074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:36.369692 (+ 618us) server_negotiation.cc:407] Connection header received
0619 13:13:36.369929 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:36.369935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:36.370003 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:36.370070 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:36.370686 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.371376 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.373146 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.373886 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.373982 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.374073 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:36.374422 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 13:13:36.374502 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:13:36.374876 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.368871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34723 (local address 127.0.97.1:36261)
0619 13:13:36.369588 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:36.369605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:36.369750 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:36.370103 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:36.370107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:36.370129 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:36.370552 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:36.370561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.371494 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.371507 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:36.372997 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:36.373010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.374104 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.374110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:36.374274 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.374309 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 13:13:36.374407 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 13:13:36.374532 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":64,"thread_start_us":374,"threads_started":1}
I20260619 13:13:36.375360 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.368825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34483 (local address 127.0.97.3:45817)
0619 13:13:36.369078 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:36.369084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:36.369787 (+ 703us) server_negotiation.cc:407] Connection header received
0619 13:13:36.369946 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:36.369949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:36.370013 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:36.370093 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:36.370707 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.371404 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.373120 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.374458 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.374564 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.374654 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:36.375022 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 13:13:36.375145 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260619 13:13:36.375881 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.368936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45817 (local address 127.0.97.1:34483)
0619 13:13:36.369692 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:36.369705 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:36.369817 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:36.370115 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:36.370119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:36.370139 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:36.370554 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:36.370562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.371526 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.371529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:36.372978 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:36.372992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.374661 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.374666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:36.374833 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.374868 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 13:13:36.375525 (+ 657us) client_negotiation.cc:240] Negotiation successful
0619 13:13:36.375636 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":15,"thread_start_us":396,"threads_started":1}
I20260619 13:13:36.375831 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d06dbe1b65243c1ab327d14057e078d" candidate_uuid: "4e311c022bd34908b4dd33e44d9abbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecd65726659457d81db587f75d3d67f" is_pre_election: true
I20260619 13:13:36.376330 1241 raft_consensus.cc:2454] T 2d06dbe1b65243c1ab327d14057e078d P 2ecd65726659457d81db587f75d3d67f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e311c022bd34908b4dd33e44d9abbba in term 0.
I20260619 13:13:36.376839 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d06dbe1b65243c1ab327d14057e078d" candidate_uuid: "4e311c022bd34908b4dd33e44d9abbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c63ec9c4a04ff78ee66b17f83411cc" is_pre_election: true
I20260619 13:13:36.376972 1042 leader_election.cc:302] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecd65726659457d81db587f75d3d67f, 4e311c022bd34908b4dd33e44d9abbba; no voters:
I20260619 13:13:36.377312 1445 raft_consensus.cc:2781] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:13:36.377316 1374 raft_consensus.cc:2454] T 2d06dbe1b65243c1ab327d14057e078d P 83c63ec9c4a04ff78ee66b17f83411cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e311c022bd34908b4dd33e44d9abbba in term 0.
I20260619 13:13:36.377455 1445 raft_consensus.cc:482] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:13:36.377552 1445 raft_consensus.cc:3037] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:36.379822 1445 raft_consensus.cc:504] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c63ec9c4a04ff78ee66b17f83411cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45817 } } peers { permanent_uuid: "4e311c022bd34908b4dd33e44d9abbba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36311 } } peers { permanent_uuid: "2ecd65726659457d81db587f75d3d67f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34723 } }
I20260619 13:13:36.380395 1445 leader_election.cc:288] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [CANDIDATE]: Term 1 election: Requested vote from peers 83c63ec9c4a04ff78ee66b17f83411cc (127.0.97.3:45817), 2ecd65726659457d81db587f75d3d67f (127.0.97.2:34723)
I20260619 13:13:36.380762 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d06dbe1b65243c1ab327d14057e078d" candidate_uuid: "4e311c022bd34908b4dd33e44d9abbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c63ec9c4a04ff78ee66b17f83411cc"
I20260619 13:13:36.380843 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d06dbe1b65243c1ab327d14057e078d" candidate_uuid: "4e311c022bd34908b4dd33e44d9abbba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecd65726659457d81db587f75d3d67f"
I20260619 13:13:36.380952 1374 raft_consensus.cc:3037] T 2d06dbe1b65243c1ab327d14057e078d P 83c63ec9c4a04ff78ee66b17f83411cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:36.381029 1241 raft_consensus.cc:3037] T 2d06dbe1b65243c1ab327d14057e078d P 2ecd65726659457d81db587f75d3d67f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:36.384008 1241 raft_consensus.cc:2454] T 2d06dbe1b65243c1ab327d14057e078d P 2ecd65726659457d81db587f75d3d67f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e311c022bd34908b4dd33e44d9abbba in term 1.
I20260619 13:13:36.384007 1374 raft_consensus.cc:2454] T 2d06dbe1b65243c1ab327d14057e078d P 83c63ec9c4a04ff78ee66b17f83411cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e311c022bd34908b4dd33e44d9abbba in term 1.
I20260619 13:13:36.384358 1042 leader_election.cc:302] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecd65726659457d81db587f75d3d67f, 4e311c022bd34908b4dd33e44d9abbba; no voters:
I20260619 13:13:36.384627 1445 raft_consensus.cc:2781] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:36.385005 1445 raft_consensus.cc:686] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [term 1 LEADER]: Becoming Leader. State: Replica: 4e311c022bd34908b4dd33e44d9abbba, State: Running, Role: LEADER
I20260619 13:13:36.385469 1445 consensus_queue.cc:237] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c63ec9c4a04ff78ee66b17f83411cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45817 } } peers { permanent_uuid: "4e311c022bd34908b4dd33e44d9abbba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36311 } } peers { permanent_uuid: "2ecd65726659457d81db587f75d3d67f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34723 } }
I20260619 13:13:36.389473 963 catalog_manager.cc:5495] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e311c022bd34908b4dd33e44d9abbba (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4e311c022bd34908b4dd33e44d9abbba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c63ec9c4a04ff78ee66b17f83411cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e311c022bd34908b4dd33e44d9abbba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ecd65726659457d81db587f75d3d67f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34723 } health_report { overall_health: UNKNOWN } } }
W20260619 13:13:36.399008 1420 tablet.cc:2369] T 2d06dbe1b65243c1ab327d14057e078d P 83c63ec9c4a04ff78ee66b17f83411cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 13:13:36 dist-test-slave-mkt3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:13:36.431166 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.416900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60512 (local address 127.0.97.1:36311)
0619 13:13:36.417064 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:36.417092 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:36.417528 (+ 436us) server_negotiation.cc:407] Connection header received
0619 13:13:36.417712 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:36.417718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:36.417792 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:36.417902 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:36.419278 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.419963 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.421172 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.421572 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.421682 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.425446 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:36.425462 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:36.425478 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:36.425516 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:36.428601 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.429244 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.429252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.429254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.429410 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.429748 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.429752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.429754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.430122 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:36.430382 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:36.430654 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 13:13:36.430863 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 13:13:36.443863 1287 tablet.cc:2369] T 2d06dbe1b65243c1ab327d14057e078d P 2ecd65726659457d81db587f75d3d67f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 13:13:36 dist-test-slave-mkt3 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 13:13:36 dist-test-slave-mkt3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:13:36.471561 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.458595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.0.97.1:36311)
0619 13:13:36.458796 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:36.458803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:36.459097 (+ 294us) server_negotiation.cc:407] Connection header received
0619 13:13:36.459274 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:36.459279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:36.459340 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:36.459418 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:36.460618 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.461327 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.462594 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.462986 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.463093 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.466495 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:36.466500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:36.466503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:36.466537 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:36.468714 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.469301 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.469307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.469310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.469449 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.469824 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.469828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.469830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.470135 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:36.470382 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:36.470626 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 13:13:36.470854 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 13:13:36 dist-test-slave-mkt3 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260619 13:13:36.492233 1153 tablet.cc:2369] T 2d06dbe1b65243c1ab327d14057e078d P 4e311c022bd34908b4dd33e44d9abbba: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 13:13:36 dist-test-slave-mkt3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:13:36.509056 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.497089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.0.97.1:36311)
0619 13:13:36.497271 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:36.497278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:36.497542 (+ 264us) server_negotiation.cc:407] Connection header received
0619 13:13:36.497716 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:36.497720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:36.497800 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:36.497924 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:36.499198 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.499970 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.501154 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.501548 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.501632 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.504943 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:36.504948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:36.504952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:36.504991 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:36.507203 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.507747 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.507751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.507753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.507859 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.508093 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.508095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.508097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.508326 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:36.508503 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:36.508675 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 13:13:36.508823 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 13:13:36.510677 1067 tablet_service.cc:2908] Rejecting scan request for tablet 2d06dbe1b65243c1ab327d14057e078d: Uninitialized: clean time has not yet been initialized
I20260619 13:13:36.513440 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 1024
I20260619 13:13:36.524796 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.514219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.0.97.1:36311)
0619 13:13:36.514387 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:36.514393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:36.514677 (+ 284us) server_negotiation.cc:407] Connection header received
0619 13:13:36.514795 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:36.514798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:36.514862 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:36.514934 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:36.515945 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.516593 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.517708 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.518095 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.518180 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.520427 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:36.520430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:36.520433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:36.520455 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:36.522613 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.523158 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.523162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.523165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.523286 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.523567 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.523570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.523572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.523810 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:36.523969 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:36.524280 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 13:13:36.524475 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:13:36.525380 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:13:36.613488 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 1024
I20260619 13:13:36.627455 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 1158
Jun 19 13:13:36 dist-test-slave-mkt3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:13:36.640779 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.628132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45388 (local address 127.0.97.2:34723)
0619 13:13:36.628287 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:36.628293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:36.628313 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:13:36.628497 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:36.628501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:36.628583 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:36.628664 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:36.629762 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.630527 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.631741 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.632121 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.632225 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.635891 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:36.635903 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:36.635917 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:36.635952 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:36.638331 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.638948 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.638964 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.638966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.639085 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.639416 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.639419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.639421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.639766 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:36.640044 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:36.640277 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 13:13:36.640468 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 13:13:36.641362 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:13:36.724875 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 1158
I20260619 13:13:36.737237 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 1291
Jun 19 13:13:36 dist-test-slave-mkt3 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:13:36.750439 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:36.737953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.0.97.3:45817)
0619 13:13:36.738107 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:36.738111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:36.738261 (+ 150us) server_negotiation.cc:407] Connection header received
0619 13:13:36.738397 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:36.738401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:36.738472 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:36.738554 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:36.739660 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.740377 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.741583 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:36.742035 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:36.742128 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:36.745748 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:36.745762 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:36.745778 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:36.745812 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:36.748349 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.748867 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.748874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.748876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.748996 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:36.749264 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:36.749266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:36.749268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:36.749579 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:36.749790 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:36.750052 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 13:13:36.750208 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 13:13:36.751027 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:13:36.835036 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 1291
I20260619 13:13:36.847463 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 931
I20260619 13:13:36.848482 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:13:36.951113 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 931
2026-06-19T13:13:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3808 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:13:37 dist-test-slave-mkt3 krb5kdc[1467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:13:37 dist-test-slave-mkt3 krb5kdc[1467](info): set up 2 sockets
Jun 19 13:13:37 dist-test-slave-mkt3 krb5kdc[1467](info): commencing operation
krb5kdc: starting...
W20260619 13:13:37.945508 388 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:13:38 dist-test-slave-mkt3 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:13:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:13:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:13:38.102640 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39039
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:38.402079 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:38.402395 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:38.402525 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:38.409667 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:13:38.409785 1480 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 13:13:38.409871 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:38.409956 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:38.410022 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:13:38.410101 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:13:38.419977 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39039
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:38.421321 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:38.422698 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:38.433748 1485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:38.433712 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:38.433789 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:38.435114 1480 server_base.cc:1034] running on GCE node
I20260619 13:13:38.436045 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:38.437446 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:38.438664 1480 hybrid_clock.cc:648] HybridClock initialized: now 1781874818438598 us; error 41 us; skew 500 ppm
I20260619 13:13:38.438987 1480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:38 dist-test-slave-mkt3 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:38.442075 1480 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:13:38.443356 1480 webserver.cc:460] Webserver started at http://127.0.97.62:40737/ using document root <none> and password file <none>
I20260619 13:13:38.444000 1480 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:38.444078 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:38.444339 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:38.447115 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "5a07e43a7aee4543936faee342ddaef3"
format_stamp: "Formatted at 2026-06-19 13:13:38 on dist-test-slave-mkt3"
server_key: "ab820beac1169c5f24ea8333f7caf8cb"
server_key_iv: "7d5b971d5fa295729136081143a21dcc"
server_key_version: "encryptionkey@0"
I20260619 13:13:38.447810 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "5a07e43a7aee4543936faee342ddaef3"
format_stamp: "Formatted at 2026-06-19 13:13:38 on dist-test-slave-mkt3"
server_key: "ab820beac1169c5f24ea8333f7caf8cb"
server_key_iv: "7d5b971d5fa295729136081143a21dcc"
server_key_version: "encryptionkey@0"
I20260619 13:13:38.452382 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:13:38.455570 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:38.456944 1480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:13:38.457079 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "5a07e43a7aee4543936faee342ddaef3"
format_stamp: "Formatted at 2026-06-19 13:13:38 on dist-test-slave-mkt3"
server_key: "ab820beac1169c5f24ea8333f7caf8cb"
server_key_iv: "7d5b971d5fa295729136081143a21dcc"
server_key_version: "encryptionkey@0"
I20260619 13:13:38.457252 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:38.475394 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:38.479431 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:38.479641 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:38.498600 1480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39039
I20260619 13:13:38.498620 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39039 every 8 connection(s)
I20260619 13:13:38.499861 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:13:38.503432 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:38.507210 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 1480
I20260619 13:13:38.507411 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:13:38.508000 388 external_mini_cluster.cc:1428] Setting key 81a821c0eb3cb6750ec0a919dde0d2e1
I20260619 13:13:38.513762 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3: Bootstrap starting.
I20260619 13:13:38.517143 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:38.518245 1557 log.cc:824] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:38.521385 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3: No bootstrap required, opened a new log
Jun 19 13:13:38 dist-test-slave-mkt3 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:38.528812 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a07e43a7aee4543936faee342ddaef3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39039 } }
I20260619 13:13:38.529155 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:38.529255 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a07e43a7aee4543936faee342ddaef3, State: Initialized, Role: FOLLOWER
I20260619 13:13:38.529806 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:38.509729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.0.97.62:39039)
0619 13:13:38.510433 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:38.510444 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:38.510489 (+ 45us) server_negotiation.cc:407] Connection header received
0619 13:13:38.512366 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:38.512386 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:38.512755 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:38.513050 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:38.514689 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:38.515982 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:38.517637 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:38.518345 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:38.518494 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:38.522899 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:38.522909 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:38.522927 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:38.522963 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:38.525865 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:38.526502 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:38.526516 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:38.526523 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:38.526694 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:38.527043 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:38.527046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:38.527048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:38.527595 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:38.527864 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:38.528281 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 13:13:38.528681 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":313,"threads_started":1}
I20260619 13:13:38.529825 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a07e43a7aee4543936faee342ddaef3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39039 } }
I20260619 13:13:38.530138 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:13:38.530241 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:13:38.530365 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:38.532641 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a07e43a7aee4543936faee342ddaef3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39039 } }
I20260619 13:13:38.533087 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a07e43a7aee4543936faee342ddaef3; no voters:
I20260619 13:13:38.533581 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:13:38.533704 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:38.534634 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [term 1 LEADER]: Becoming Leader. State: Replica: 5a07e43a7aee4543936faee342ddaef3, State: Running, Role: LEADER
I20260619 13:13:38.535099 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a07e43a7aee4543936faee342ddaef3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39039 } }
I20260619 13:13:38.535528 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:13:38.537521 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a07e43a7aee4543936faee342ddaef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a07e43a7aee4543936faee342ddaef3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39039 } } }
I20260619 13:13:38.537997 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:38.538784 1571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:13:38.538650 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a07e43a7aee4543936faee342ddaef3. Latest consensus state: current_term: 1 leader_uuid: "5a07e43a7aee4543936faee342ddaef3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a07e43a7aee4543936faee342ddaef3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39039 } } }
I20260619 13:13:38.538928 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3 [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:38.544072 1571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:13:38.551959 1571 catalog_manager.cc:1347] Generated new cluster ID: 11b748b7c4794fd0ac555b968825d52a
I20260619 13:13:38.552050 1571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:13:38.562804 1571 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:13:38.564589 1571 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:13:38.578696 1571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3: Generated new TSK 0
I20260619 13:13:38.579588 1571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:13:38.632617 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39039
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:38.954537 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:38.954880 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:38.955076 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:38.962749 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:38.962848 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:38.962937 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:13:38.963014 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:13:38.972512 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:38.973654 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:38.974982 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:38.987161 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:38.987110 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:38.987342 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:38.988236 1583 server_base.cc:1034] running on GCE node
I20260619 13:13:38.988747 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:38.989643 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:38.990844 1583 hybrid_clock.cc:648] HybridClock initialized: now 1781874818990774 us; error 47 us; skew 500 ppm
I20260619 13:13:38.991117 1583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:38 dist-test-slave-mkt3 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:38.993997 1583 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:13:38.995339 1583 webserver.cc:460] Webserver started at http://127.0.97.1:46289/ using document root <none> and password file <none>
I20260619 13:13:38.996004 1583 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:38.996101 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:38.996394 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:38.999204 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "b0df3db2d34e4367b60d855569f306f4"
format_stamp: "Formatted at 2026-06-19 13:13:38 on dist-test-slave-mkt3"
server_key: "b5d21a07364f98b35ede161edcf01a34"
server_key_iv: "8c6719384f57822d5ea06648bae1293a"
server_key_version: "encryptionkey@0"
I20260619 13:13:38.999975 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b0df3db2d34e4367b60d855569f306f4"
format_stamp: "Formatted at 2026-06-19 13:13:38 on dist-test-slave-mkt3"
server_key: "b5d21a07364f98b35ede161edcf01a34"
server_key_iv: "8c6719384f57822d5ea06648bae1293a"
server_key_version: "encryptionkey@0"
I20260619 13:13:39.004979 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:13:39.008337 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:39.009660 1583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:13:39.009807 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "b0df3db2d34e4367b60d855569f306f4"
format_stamp: "Formatted at 2026-06-19 13:13:38 on dist-test-slave-mkt3"
server_key: "b5d21a07364f98b35ede161edcf01a34"
server_key_iv: "8c6719384f57822d5ea06648bae1293a"
server_key_version: "encryptionkey@0"
I20260619 13:13:39.010031 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:39.043593 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:39.048194 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:39.048441 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:39.051052 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:39.051136 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:39.051259 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:39.051365 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 13:13:39 dist-test-slave-mkt3 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:39.076179 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:39.051529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37097 (local address 127.0.97.62:39039)
0619 13:13:39.058180 (+ 6651us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:39.058185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:39.058204 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:13:39.058268 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:39.058274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:39.058357 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:39.058476 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:39.060929 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.061987 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.064507 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.064904 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.064997 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:39.069671 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:39.069679 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:39.069684 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:39.069726 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:39.072808 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:39.073738 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:39.073743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:39.073747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:39.073944 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:39.074434 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:39.074440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:39.074444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:39.074752 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:39.074875 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:39.075598 (+ 723us) server_negotiation.cc:299] Negotiation successful
0619 13:13:39.075809 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6551,"thread_start_us":6453,"threads_started":1}
I20260619 13:13:39.077194 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:39.052224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39039 (local address 127.0.97.1:37097)
0619 13:13:39.053161 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:39.053228 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:39.055742 (+ 2514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:39.058805 (+ 3063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:39.058819 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:39.059693 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:39.060732 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:39.060761 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.062167 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.062173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:39.064366 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:39.064383 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.065127 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.065133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:39.065468 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:39.066239 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:39.066275 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:39.069427 (+ 3152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:39.072941 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:39.072951 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:39.072972 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:39.073601 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:39.074086 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:39.074091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:39.074094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:39.074316 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:39.074882 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:39.074891 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:39.075260 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 13:13:39.075493 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 13:13:39.076009 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":378,"threads_started":1}
I20260619 13:13:39.100852 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40843
I20260619 13:13:39.100874 1718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40843 every 8 connection(s)
I20260619 13:13:39.102272 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:13:39.102392 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:39.082438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55257 (local address 127.0.97.62:39039)
0619 13:13:39.082611 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:39.082618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:39.090476 (+ 7858us) server_negotiation.cc:407] Connection header received
0619 13:13:39.090667 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:39.090672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:39.090754 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:39.090863 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:39.091809 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.092743 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.094735 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.095241 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.095349 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:39.097066 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:39.097071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:39.097075 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:39.097104 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:39.099686 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:39.100515 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:39.100520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:39.100523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:39.100653 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:39.100996 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:39.101011 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:39.101013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:39.101288 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:39.101422 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:39.101808 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 13:13:39.102032 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:13:39.102517 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:39.082524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39039 (local address 127.0.97.1:55257)
0619 13:13:39.090351 (+ 7827us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:39.090375 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:39.090538 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:39.090886 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:39.090889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:39.091353 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:39.091667 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:39.091679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.092901 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.092907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:39.094538 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:39.094558 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.095414 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.095422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:39.095558 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:39.096187 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:39.096203 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:39.096892 (+ 689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:39.099805 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:39.099814 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:39.099818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:39.100399 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:39.100756 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:39.100762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:39.100766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:39.100900 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:39.101398 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:39.101403 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:39.101574 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 13:13:39.101702 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 13:13:39.102036 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7677,"thread_start_us":7525,"threads_started":1}
I20260619 13:13:39.104322 1719 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39039
I20260619 13:13:39.104565 1719 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:39.105337 1719 heartbeater.cc:511] Master 127.0.97.62:39039 requested a full tablet report, sending...
I20260619 13:13:39.107141 1512 ts_manager.cc:194] Registered new tserver with Master: b0df3db2d34e4367b60d855569f306f4 (127.0.97.1:40843)
I20260619 13:13:39.108470 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37097
I20260619 13:13:39.110054 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 1583
I20260619 13:13:39.110257 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:13:39.111009 388 external_mini_cluster.cc:1428] Setting key 9ff8302d1c65b29974f43c34f6da301e
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:13:39.144088 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39039
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:39.443588 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:39.443869 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:39.444007 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:39.450953 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:39.451069 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:39.451205 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:13:39.451314 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:13:39.460877 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:39.462141 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:39.463398 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:39.475250 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:39.475267 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:39.475250 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:39.476310 1723 server_base.cc:1034] running on GCE node
I20260619 13:13:39.476858 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:39.477679 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:39.478878 1723 hybrid_clock.cc:648] HybridClock initialized: now 1781874819478809 us; error 41 us; skew 500 ppm
I20260619 13:13:39.479158 1723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:39 dist-test-slave-mkt3 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:39.482297 1723 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:13:39.483829 1723 webserver.cc:460] Webserver started at http://127.0.97.2:36451/ using document root <none> and password file <none>
I20260619 13:13:39.484500 1723 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:39.484586 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:39.484872 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:39.487725 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "1fe03a3a9f034050bd4794ee263556c0"
format_stamp: "Formatted at 2026-06-19 13:13:39 on dist-test-slave-mkt3"
server_key: "57c6e69ca8fa0634f26622e666cd0a64"
server_key_iv: "2efcececc7becb134878b9fc1d7d396d"
server_key_version: "encryptionkey@0"
I20260619 13:13:39.488476 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1fe03a3a9f034050bd4794ee263556c0"
format_stamp: "Formatted at 2026-06-19 13:13:39 on dist-test-slave-mkt3"
server_key: "57c6e69ca8fa0634f26622e666cd0a64"
server_key_iv: "2efcececc7becb134878b9fc1d7d396d"
server_key_version: "encryptionkey@0"
I20260619 13:13:39.493367 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 13:13:39.496743 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:39.498131 1723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:13:39.498268 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "1fe03a3a9f034050bd4794ee263556c0"
format_stamp: "Formatted at 2026-06-19 13:13:39 on dist-test-slave-mkt3"
server_key: "57c6e69ca8fa0634f26622e666cd0a64"
server_key_iv: "2efcececc7becb134878b9fc1d7d396d"
server_key_version: "encryptionkey@0"
I20260619 13:13:39.498461 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:39.520480 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:39.524108 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:39.524333 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:39.526789 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:39.526885 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:39.527019 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:39.527096 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:13:39 dist-test-slave-mkt3 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:39.550060 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:39.527189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34547 (local address 127.0.97.62:39039)
0619 13:13:39.527346 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:39.527352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:39.528705 (+ 1353us) server_negotiation.cc:407] Connection header received
0619 13:13:39.530655 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:39.530661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:39.530738 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:39.530817 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:39.532758 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.533759 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.536654 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.537161 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.537280 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:39.542418 (+ 5138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:39.542427 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:39.542432 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:39.542481 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:39.545634 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:39.546686 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:39.546693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:39.546696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:39.546845 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:39.547420 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:39.547425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:39.547428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:39.547782 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:39.549358 (+ 1576us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:39.549510 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 13:13:39.549725 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 13:13:39.551154 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:39.527828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39039 (local address 127.0.97.2:34547)
0619 13:13:39.528557 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:39.528601 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:39.530387 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:39.531041 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:39.531048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:39.531715 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:39.532586 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:39.532615 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.533927 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.533933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:39.536439 (+ 2506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:39.536458 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.537418 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.537429 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:39.537762 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:39.538566 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:39.538609 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:39.542118 (+ 3509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:39.545812 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:39.545823 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:39.545868 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:39.546549 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:39.546970 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:39.546976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:39.546980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:39.547279 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:39.547950 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:39.547958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:39.548546 (+ 588us) client_negotiation.cc:769] Sending connection context
0619 13:13:39.548896 (+ 350us) client_negotiation.cc:240] Negotiation successful
0619 13:13:39.549508 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":319,"threads_started":1}
I20260619 13:13:39.562592 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42773
I20260619 13:13:39.562750 1857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42773 every 8 connection(s)
I20260619 13:13:39.563898 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:13:39.565908 1858 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39039
I20260619 13:13:39.566116 1858 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:39.566751 1858 heartbeater.cc:511] Master 127.0.97.62:39039 requested a full tablet report, sending...
I20260619 13:13:39.568084 1512 ts_manager.cc:194] Registered new tserver with Master: 1fe03a3a9f034050bd4794ee263556c0 (127.0.97.2:42773)
I20260619 13:13:39.568516 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 1723
I20260619 13:13:39.568709 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:13:39.569041 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34547
I20260619 13:13:39.569232 388 external_mini_cluster.cc:1428] Setting key 7decccb682d02c1ed84c08cc4ce7204e
I20260619 13:13:39.570264 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:39.557372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43763 (local address 127.0.97.62:39039)
0619 13:13:39.557551 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:39.557558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:39.558067 (+ 509us) server_negotiation.cc:407] Connection header received
0619 13:13:39.558290 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:39.558296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:39.558361 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:39.558433 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:39.559714 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.560673 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.562298 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.562894 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.563015 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:39.564752 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:39.564759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:39.564764 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:39.564800 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:39.567306 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:39.567976 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:39.567981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:39.567984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:39.568156 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:39.568560 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:39.568566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:39.568570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:39.569024 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:39.569177 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:39.569706 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 13:13:39.569934 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:13:39.570345 1843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:39.557330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39039 (local address 127.0.97.2:43763)
0619 13:13:39.557964 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:39.557985 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:39.558178 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:39.558478 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:39.558482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:39.559059 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:39.559480 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:39.559530 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.560876 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.560885 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:39.562176 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:39.562187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.563069 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.563079 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:39.563231 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:39.563772 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:39.563787 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:39.564537 (+ 750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:39.567417 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:39.567422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:39.567426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:39.567878 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:39.568291 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:39.568295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:39.568298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:39.568431 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:39.569272 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:39.569277 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:39.569450 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 13:13:39.569600 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 13:13:39.569921 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":337,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:13:39.602309 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39039
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:39.903244 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:39.903506 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:39.903635 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:39.910630 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:39.910730 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:39.910845 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:13:39.910944 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:13:39.920362 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:39.921612 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:39.922897 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:39.934777 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:39.934948 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:39.934909 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:39.935678 1862 server_base.cc:1034] running on GCE node
I20260619 13:13:39.936223 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:39.937163 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:39.938352 1862 hybrid_clock.cc:648] HybridClock initialized: now 1781874819938296 us; error 32 us; skew 500 ppm
I20260619 13:13:39.938673 1862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:39 dist-test-slave-mkt3 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:39.941756 1862 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:13:39.943037 1862 webserver.cc:460] Webserver started at http://127.0.97.3:44349/ using document root <none> and password file <none>
I20260619 13:13:39.943676 1862 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:39.943763 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:39.944072 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:39.946852 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "3383208265e648d9bea3bbc3222d5e86"
format_stamp: "Formatted at 2026-06-19 13:13:39 on dist-test-slave-mkt3"
server_key: "4fc712219e3d876d04797b43d2b3ade9"
server_key_iv: "06f9a1a1e6e14469b477fa0935b5c0df"
server_key_version: "encryptionkey@0"
I20260619 13:13:39.947576 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3383208265e648d9bea3bbc3222d5e86"
format_stamp: "Formatted at 2026-06-19 13:13:39 on dist-test-slave-mkt3"
server_key: "4fc712219e3d876d04797b43d2b3ade9"
server_key_iv: "06f9a1a1e6e14469b477fa0935b5c0df"
server_key_version: "encryptionkey@0"
I20260619 13:13:39.952549 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:13:39.955781 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:39.957010 1862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:13:39.957152 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "3383208265e648d9bea3bbc3222d5e86"
format_stamp: "Formatted at 2026-06-19 13:13:39 on dist-test-slave-mkt3"
server_key: "4fc712219e3d876d04797b43d2b3ade9"
server_key_iv: "06f9a1a1e6e14469b477fa0935b5c0df"
server_key_version: "encryptionkey@0"
I20260619 13:13:39.957324 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:39.981760 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:39.985941 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:39.986191 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:39.988679 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:39.988778 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:39.988897 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:39.988961 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:13:40 dist-test-slave-mkt3 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:40.010171 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:39.989291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56887 (local address 127.0.97.62:39039)
0619 13:13:39.989471 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:39.989477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:39.990893 (+ 1416us) server_negotiation.cc:407] Connection header received
0619 13:13:39.992780 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:39.992785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:39.992862 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:39.992946 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:39.995033 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.996177 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.998547 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.999165 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.999300 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.003571 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:40.003579 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:40.003585 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:40.003637 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:40.007027 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:40.007751 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:40.007757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:40.007761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:40.007923 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:40.008360 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:40.008365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:40.008369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:40.008719 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:40.008850 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:40.009586 (+ 736us) server_negotiation.cc:299] Negotiation successful
0619 13:13:40.009795 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:13:40.011487 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:39.989978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39039 (local address 127.0.97.3:56887)
0619 13:13:39.990746 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:39.990785 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:39.992507 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:39.993208 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:39.993221 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:39.993986 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:39.994803 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:39.994825 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.996381 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.996387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:39.998399 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:39.998411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:39.999419 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:39.999425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:39.999672 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.000428 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:40.000459 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:40.003329 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:40.007197 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:40.007204 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:40.007216 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:40.007632 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:40.008020 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:40.008023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:40.008026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:40.008223 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:40.008842 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:40.008853 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:40.009186 (+ 333us) client_negotiation.cc:769] Sending connection context
0619 13:13:40.009415 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 13:13:40.009932 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":322,"threads_started":1}
I20260619 13:13:40.033412 1996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43905 every 8 connection(s)
I20260619 13:13:40.033408 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43905
I20260619 13:13:40.034966 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:13:40.036640 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 1862
I20260619 13:13:40.036840 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:13:40.037046 1997 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39039
I20260619 13:13:40.037349 388 external_mini_cluster.cc:1428] Setting key 65ed380bb417ad472e535169f89987c3
I20260619 13:13:40.037550 1997 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:40.038287 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.024745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52101 (local address 127.0.97.62:39039)
0619 13:13:40.024931 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:40.024938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:40.025539 (+ 601us) server_negotiation.cc:407] Connection header received
0619 13:13:40.025670 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:40.025676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:40.025763 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:40.025903 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:40.026893 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.028180 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.029457 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.030154 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.030278 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.032147 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:40.032153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:40.032158 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:40.032195 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:40.035311 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:40.036164 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:40.036170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:40.036173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:40.036318 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:40.036777 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:40.036782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:40.036785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:40.037139 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:40.037270 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:40.037745 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 13:13:40.037968 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 13:13:40.038368 1987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.024871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39039 (local address 127.0.97.3:52101)
0619 13:13:40.025438 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:40.025462 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:40.025574 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:40.025961 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:40.025966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:40.026477 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:40.026755 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.026765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.028338 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.028342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.029336 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.029346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.030355 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.030362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.030558 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.031224 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:40.031255 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:40.031967 (+ 712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:40.035451 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:40.035457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:40.035461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:40.036019 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:40.036455 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:40.036460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:40.036463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:40.036608 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:40.037265 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:40.037269 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:40.037445 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 13:13:40.037616 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 13:13:40.037935 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":309,"threads_started":1}
I20260619 13:13:40.038559 1997 heartbeater.cc:511] Master 127.0.97.62:39039 requested a full tablet report, sending...
I20260619 13:13:40.039942 1512 ts_manager.cc:194] Registered new tserver with Master: 3383208265e648d9bea3bbc3222d5e86 (127.0.97.3:43905)
I20260619 13:13:40.040923 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56887
I20260619 13:13:40.052323 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:13:40 dist-test-slave-mkt3 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 13:13:40 dist-test-slave-mkt3 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:40.092514 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.079854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.0.97.62:39039)
0619 13:13:40.080002 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:40.080007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:40.080347 (+ 340us) server_negotiation.cc:407] Connection header received
0619 13:13:40.080490 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:40.080494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:40.080570 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:40.080663 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:40.081974 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.082671 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.083840 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.084256 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.084387 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.087888 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:40.087893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:40.087897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:40.087929 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:40.090230 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:40.090788 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:40.090793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:40.090795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:40.090912 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:40.091243 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:40.091248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:40.091252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:40.091488 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:40.091694 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:40.091963 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 13:13:40.092187 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 13:13:40.096323 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:13:40.102916 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:13:40.111462 1719 heartbeater.cc:503] Master 127.0.97.62:39039 was elected leader, sending a full tablet report...
I20260619 13:13:40.123152 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.113702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.0.97.2:42773)
0619 13:13:40.114285 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:40.114294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:40.115089 (+ 795us) server_negotiation.cc:407] Connection header received
0619 13:13:40.115458 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:40.115467 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:40.115748 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:40.115902 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:40.116890 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.118366 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.120727 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.122061 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.122178 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.122370 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:40.122707 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 13:13:40.122869 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":779,"threads_started":1}
I20260619 13:13:40.123207 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.113501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42773 (local address 127.0.0.1:48204)
0619 13:13:40.114943 (+ 1442us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:40.114983 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:40.115237 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:40.116184 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:40.116192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:40.116223 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:40.116695 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.116708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.118508 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.118513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.120590 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.120608 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.122242 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.122249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.122450 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.122500 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 13:13:40.122652 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 13:13:40.122822 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"mutex_wait_us":25,"thread_start_us":932,"threads_started":1}
I20260619 13:13:40.124440 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.114290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.0.97.3:43905)
0619 13:13:40.117297 (+ 3007us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:40.117303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:40.117327 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:13:40.117393 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:40.117398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:40.117552 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:40.117775 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:40.118429 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.119744 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.122009 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.123323 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.123437 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.123633 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:40.124007 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 13:13:40.124156 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2883,"thread_start_us":2717,"threads_started":1}
I20260619 13:13:40.125301 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.116260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36228 (local address 127.0.97.1:40843)
0619 13:13:40.118355 (+ 2095us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:40.118362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:40.118385 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:13:40.118458 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:40.118465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:40.118677 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:40.118858 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:40.119470 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.120486 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.122785 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.124084 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.124209 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.124382 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:40.124887 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 13:13:40.125040 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1957,"thread_start_us":2233,"threads_started":1}
I20260619 13:13:40.125295 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.115854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40843 (local address 127.0.0.1:36228)
0619 13:13:40.116303 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:40.116319 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:40.116424 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:40.118857 (+ 2433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:40.118861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:40.118885 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:40.119298 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.119311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.120611 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.120620 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.122618 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.122632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.124491 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.124497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.124736 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.124759 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:13:40.124835 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 13:13:40.124953 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":20,"thread_start_us":551,"threads_started":1}
I20260619 13:13:40.126819 1788 tablet_service.cc:1467] Processing CreateTablet for tablet df467930f34d4a478356d9433fd07e0f (DEFAULT_TABLE table=test-table [id=94977c736a22493eb66c06d13210dcfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:40.127928 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df467930f34d4a478356d9433fd07e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:40.128100 1649 tablet_service.cc:1467] Processing CreateTablet for tablet df467930f34d4a478356d9433fd07e0f (DEFAULT_TABLE table=test-table [id=94977c736a22493eb66c06d13210dcfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:40.129102 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df467930f34d4a478356d9433fd07e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:40.133955 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.113624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43905 (local address 127.0.0.1:57496)
0619 13:13:40.115714 (+ 2090us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:40.115729 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:40.115829 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:40.117772 (+ 1943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:40.117776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:40.117798 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:40.118230 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.118242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.119873 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.119879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.121858 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.121871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.123739 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.123744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.123845 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.123865 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 13:13:40.123932 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 13:13:40.124047 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1968,"mutex_wait_us":1,"thread_start_us":1896,"threads_started":1}
I20260619 13:13:40.137395 1927 tablet_service.cc:1467] Processing CreateTablet for tablet df467930f34d4a478356d9433fd07e0f (DEFAULT_TABLE table=test-table [id=94977c736a22493eb66c06d13210dcfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:40.138756 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df467930f34d4a478356d9433fd07e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:40.139067 2016 tablet_bootstrap.cc:492] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0: Bootstrap starting.
I20260619 13:13:40.140761 2017 tablet_bootstrap.cc:492] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4: Bootstrap starting.
I20260619 13:13:40.142431 2016 tablet_bootstrap.cc:654] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:40.143996 2017 tablet_bootstrap.cc:654] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:40.143986 2016 log.cc:824] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:40.144999 2017 log.cc:824] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:40.146723 2016 tablet_bootstrap.cc:492] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0: No bootstrap required, opened a new log
I20260619 13:13:40.146983 2016 ts_tablet_manager.cc:1397] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:13:40.147682 2017 tablet_bootstrap.cc:492] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4: No bootstrap required, opened a new log
I20260619 13:13:40.147928 2017 ts_tablet_manager.cc:1397] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260619 13:13:40.150660 2020 tablet_bootstrap.cc:492] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86: Bootstrap starting.
I20260619 13:13:40.153388 2020 tablet_bootstrap.cc:654] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:40.154331 2020 log.cc:824] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:40.156543 2020 tablet_bootstrap.cc:492] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86: No bootstrap required, opened a new log
I20260619 13:13:40.156766 2020 ts_tablet_manager.cc:1397] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260619 13:13:40.157783 2016 raft_consensus.cc:348] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } }
I20260619 13:13:40.158243 2016 raft_consensus.cc:374] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:40.158015 2017 raft_consensus.cc:348] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } }
I20260619 13:13:40.158341 2016 raft_consensus.cc:729] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe03a3a9f034050bd4794ee263556c0, State: Initialized, Role: FOLLOWER
I20260619 13:13:40.158367 2017 raft_consensus.cc:374] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:40.158440 2017 raft_consensus.cc:729] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0df3db2d34e4367b60d855569f306f4, State: Initialized, Role: FOLLOWER
I20260619 13:13:40.158944 2016 consensus_queue.cc:260] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } }
I20260619 13:13:40.159014 2017 consensus_queue.cc:260] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } }
I20260619 13:13:40.160480 1858 heartbeater.cc:503] Master 127.0.97.62:39039 was elected leader, sending a full tablet report...
I20260619 13:13:40.160708 2016 ts_tablet_manager.cc:1428] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260619 13:13:40.163628 2017 ts_tablet_manager.cc:1428] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
I20260619 13:13:40.165753 2020 raft_consensus.cc:348] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } }
I20260619 13:13:40.166115 2020 raft_consensus.cc:374] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:40.166208 2020 raft_consensus.cc:729] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3383208265e648d9bea3bbc3222d5e86, State: Initialized, Role: FOLLOWER
I20260619 13:13:40.166764 2020 consensus_queue.cc:260] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } }
I20260619 13:13:40.167613 1997 heartbeater.cc:503] Master 127.0.97.62:39039 was elected leader, sending a full tablet report...
I20260619 13:13:40.168160 2020 ts_tablet_manager.cc:1428] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
W20260619 13:13:40.286609 1998 tablet.cc:2369] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:13:40.315752 1859 tablet.cc:2369] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:13:40.341656 2023 raft_consensus.cc:482] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:13:40.342027 2023 raft_consensus.cc:504] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } }
I20260619 13:13:40.343684 2023 leader_election.cc:288] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3383208265e648d9bea3bbc3222d5e86 (127.0.97.3:43905), b0df3db2d34e4367b60d855569f306f4 (127.0.97.1:40843)
I20260619 13:13:40.350159 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.344080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40513 (local address 127.0.97.3:43905)
0619 13:13:40.344245 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:40.344249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:40.344619 (+ 370us) server_negotiation.cc:407] Connection header received
0619 13:13:40.344796 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:40.344802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:40.344871 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:40.344963 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:40.345528 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.346248 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.348571 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.349303 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.349413 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.349513 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:40.349780 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 13:13:40.349921 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:13:40.350313 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.344749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44267 (local address 127.0.97.1:40843)
0619 13:13:40.344932 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:40.344937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:40.345236 (+ 299us) server_negotiation.cc:407] Connection header received
0619 13:13:40.345395 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:40.345399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:40.345461 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:40.345528 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:40.346047 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.346805 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.348740 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.349456 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.349566 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.349658 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:40.349968 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 13:13:40.350093 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 13:13:40.350324 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.343906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43905 (local address 127.0.97.2:40513)
0619 13:13:40.344482 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:40.344499 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:40.344651 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:40.345005 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:40.345010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:40.345046 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:40.345389 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.345399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.346371 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.346376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.348435 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.348446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.349492 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.349498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.349636 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.349663 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 13:13:40.349740 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 13:13:40.349917 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":20,"thread_start_us":299,"threads_started":1}
I20260619 13:13:40.350453 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.344599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40843 (local address 127.0.97.2:44267)
0619 13:13:40.345147 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:40.345160 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:40.345275 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:40.345556 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:40.345560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:40.345589 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:40.345905 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.345915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.346929 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.346933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.348602 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.348614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.349658 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.349664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.349815 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.349860 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 13:13:40.349942 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 13:13:40.350061 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":1,"thread_start_us":335,"threads_started":1}
I20260619 13:13:40.351342 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df467930f34d4a478356d9433fd07e0f" candidate_uuid: "1fe03a3a9f034050bd4794ee263556c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0df3db2d34e4367b60d855569f306f4" is_pre_election: true
I20260619 13:13:40.351346 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df467930f34d4a478356d9433fd07e0f" candidate_uuid: "1fe03a3a9f034050bd4794ee263556c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3383208265e648d9bea3bbc3222d5e86" is_pre_election: true
I20260619 13:13:40.351861 1674 raft_consensus.cc:2454] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fe03a3a9f034050bd4794ee263556c0 in term 0.
I20260619 13:13:40.351859 1948 raft_consensus.cc:2454] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fe03a3a9f034050bd4794ee263556c0 in term 0.
I20260619 13:13:40.352464 1739 leader_election.cc:302] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe03a3a9f034050bd4794ee263556c0, b0df3db2d34e4367b60d855569f306f4; no voters:
I20260619 13:13:40.352785 2023 raft_consensus.cc:2781] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:13:40.352913 2023 raft_consensus.cc:482] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:13:40.353020 2023 raft_consensus.cc:3037] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Advancing to term 1
W20260619 13:13:40.354523 1720 tablet.cc:2369] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:13:40.355455 2023 raft_consensus.cc:504] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } }
I20260619 13:13:40.355965 2023 leader_election.cc:288] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [CANDIDATE]: Term 1 election: Requested vote from peers 3383208265e648d9bea3bbc3222d5e86 (127.0.97.3:43905), b0df3db2d34e4367b60d855569f306f4 (127.0.97.1:40843)
I20260619 13:13:40.356365 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df467930f34d4a478356d9433fd07e0f" candidate_uuid: "1fe03a3a9f034050bd4794ee263556c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3383208265e648d9bea3bbc3222d5e86"
I20260619 13:13:40.356428 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df467930f34d4a478356d9433fd07e0f" candidate_uuid: "1fe03a3a9f034050bd4794ee263556c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0df3db2d34e4367b60d855569f306f4"
I20260619 13:13:40.356523 1948 raft_consensus.cc:3037] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:40.356621 1674 raft_consensus.cc:3037] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:40.359014 1948 raft_consensus.cc:2454] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe03a3a9f034050bd4794ee263556c0 in term 1.
I20260619 13:13:40.359097 1674 raft_consensus.cc:2454] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe03a3a9f034050bd4794ee263556c0 in term 1.
I20260619 13:13:40.359354 1739 leader_election.cc:302] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe03a3a9f034050bd4794ee263556c0, 3383208265e648d9bea3bbc3222d5e86; no voters:
I20260619 13:13:40.359623 2023 raft_consensus.cc:2781] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:40.360070 2023 raft_consensus.cc:686] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [term 1 LEADER]: Becoming Leader. State: Replica: 1fe03a3a9f034050bd4794ee263556c0, State: Running, Role: LEADER
I20260619 13:13:40.360584 2023 consensus_queue.cc:237] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } }
I20260619 13:13:40.365142 1512 catalog_manager.cc:5495] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fe03a3a9f034050bd4794ee263556c0 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1fe03a3a9f034050bd4794ee263556c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } health_report { overall_health: UNKNOWN } } }
I20260619 13:13:40.420902 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.416299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48210 (local address 127.0.97.2:42773)
0619 13:13:40.416483 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:40.416488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:40.416505 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:13:40.416724 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:40.416727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:40.416797 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:40.416904 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:13:40.417177 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.417915 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.419111 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.419495 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.419589 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.419991 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:13:40.420446 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:13:40.420566 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:40.420607 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 13:13:40.420665 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:13:40.428936 1674 raft_consensus.cc:1261] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4 [term 1 FOLLOWER]: Refusing update from remote peer 1fe03a3a9f034050bd4794ee263556c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:13:40.428992 1948 raft_consensus.cc:1261] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86 [term 1 FOLLOWER]: Refusing update from remote peer 1fe03a3a9f034050bd4794ee263556c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:13:40.429872 2027 consensus_queue.cc:1035] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:13:40.430231 2023 consensus_queue.cc:1035] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:13:40.457751 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:13:40.458056 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:13:40.461305 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3: Sending DeleteTablet for 3 replicas of tablet df467930f34d4a478356d9433fd07e0f
I20260619 13:13:40.462468 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet df467930f34d4a478356d9433fd07e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48204
I20260619 13:13:40.462471 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet df467930f34d4a478356d9433fd07e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36228
I20260619 13:13:40.462786 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet df467930f34d4a478356d9433fd07e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57496
I20260619 13:13:40.465953 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:13:40.466620 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:13:40.474314 2042 tablet_replica.cc:330] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86: stopping tablet replica
I20260619 13:13:40.474866 2042 raft_consensus.cc:2229] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:13:40.476239 2042 raft_consensus.cc:2258] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:40.476538 1787 tablet_service.cc:1467] Processing CreateTablet for tablet e2d5f0a4ebbe4b539749c97c1f069547 (DEFAULT_TABLE table=test-table [id=5d8d0777e86f4c2d814eb36d68136ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:40.477018 2041 tablet_replica.cc:330] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4: stopping tablet replica
I20260619 13:13:40.477159 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d5f0a4ebbe4b539749c97c1f069547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:40.477419 2041 raft_consensus.cc:2229] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:13:40.478467 2041 raft_consensus.cc:2258] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:40.479765 2042 ts_tablet_manager.cc:1905] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:40.481520 2041 ts_tablet_manager.cc:1905] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:40.484864 2016 tablet_bootstrap.cc:492] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0: Bootstrap starting.
I20260619 13:13:40.485401 1927 tablet_service.cc:1467] Processing CreateTablet for tablet e2d5f0a4ebbe4b539749c97c1f069547 (DEFAULT_TABLE table=test-table [id=5d8d0777e86f4c2d814eb36d68136ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:40.486101 1649 tablet_service.cc:1467] Processing CreateTablet for tablet e2d5f0a4ebbe4b539749c97c1f069547 (DEFAULT_TABLE table=test-table [id=5d8d0777e86f4c2d814eb36d68136ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:40.486478 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d5f0a4ebbe4b539749c97c1f069547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:40.486492 2016 tablet_bootstrap.cc:654] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:40.492336 2041 ts_tablet_manager.cc:1918] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:13:40.492727 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d5f0a4ebbe4b539749c97c1f069547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:40.493686 2041 log.cc:1196] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/df467930f34d4a478356d9433fd07e0f
I20260619 13:13:40.494401 2042 ts_tablet_manager.cc:1918] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:13:40.494529 2016 tablet_bootstrap.cc:492] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0: No bootstrap required, opened a new log
I20260619 13:13:40.494767 2016 ts_tablet_manager.cc:1397] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 13:13:40.495668 2040 tablet_replica.cc:330] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0: stopping tablet replica
I20260619 13:13:40.496203 2040 raft_consensus.cc:2229] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:13:40.496429 2042 log.cc:1196] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/df467930f34d4a478356d9433fd07e0f
I20260619 13:13:40.496969 2042 ts_tablet_manager.cc:1939] T df467930f34d4a478356d9433fd07e0f P 3383208265e648d9bea3bbc3222d5e86: Deleting consensus metadata
I20260619 13:13:40.497357 2041 ts_tablet_manager.cc:1939] T df467930f34d4a478356d9433fd07e0f P b0df3db2d34e4367b60d855569f306f4: Deleting consensus metadata
I20260619 13:13:40.496606 2016 raft_consensus.cc:348] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:40.497570 2016 raft_consensus.cc:374] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:40.497609 2040 raft_consensus.cc:2258] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:40.498610 2020 tablet_bootstrap.cc:492] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86: Bootstrap starting.
I20260619 13:13:40.500095 2020 tablet_bootstrap.cc:654] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:40.497716 2016 raft_consensus.cc:729] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe03a3a9f034050bd4794ee263556c0, State: Initialized, Role: FOLLOWER
I20260619 13:13:40.503247 1496 catalog_manager.cc:4841] TS 3383208265e648d9bea3bbc3222d5e86 (127.0.97.3:43905): tablet df467930f34d4a478356d9433fd07e0f (table test-table [id=94977c736a22493eb66c06d13210dcfa]) successfully deleted
I20260619 13:13:40.503660 2017 tablet_bootstrap.cc:492] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4: Bootstrap starting.
I20260619 13:13:40.503315 2016 consensus_queue.cc:260] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:40.505070 2017 tablet_bootstrap.cc:654] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:40.506773 2016 ts_tablet_manager.cc:1428] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20260619 13:13:40.510742 1496 catalog_manager.cc:4841] TS b0df3db2d34e4367b60d855569f306f4 (127.0.97.1:40843): tablet df467930f34d4a478356d9433fd07e0f (table test-table [id=94977c736a22493eb66c06d13210dcfa]) successfully deleted
I20260619 13:13:40.512421 2040 ts_tablet_manager.cc:1905] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:40.521426 2040 ts_tablet_manager.cc:1918] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:13:40.521595 2040 log.cc:1196] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/df467930f34d4a478356d9433fd07e0f
I20260619 13:13:40.522142 2040 ts_tablet_manager.cc:1939] T df467930f34d4a478356d9433fd07e0f P 1fe03a3a9f034050bd4794ee263556c0: Deleting consensus metadata
I20260619 13:13:40.522929 2017 tablet_bootstrap.cc:492] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4: No bootstrap required, opened a new log
I20260619 13:13:40.523082 2017 ts_tablet_manager.cc:1397] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.004s
I20260619 13:13:40.523721 1498 catalog_manager.cc:4841] TS 1fe03a3a9f034050bd4794ee263556c0 (127.0.97.2:42773): tablet df467930f34d4a478356d9433fd07e0f (table test-table [id=94977c736a22493eb66c06d13210dcfa]) successfully deleted
I20260619 13:13:40.524365 2020 tablet_bootstrap.cc:492] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86: No bootstrap required, opened a new log
I20260619 13:13:40.524318 2017 raft_consensus.cc:348] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:40.524497 2020 ts_tablet_manager.cc:1397] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.008s
I20260619 13:13:40.524551 2017 raft_consensus.cc:374] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:40.524641 2017 raft_consensus.cc:729] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0df3db2d34e4367b60d855569f306f4, State: Initialized, Role: FOLLOWER
I20260619 13:13:40.524886 2017 consensus_queue.cc:260] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:40.525116 2020 raft_consensus.cc:348] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:40.525316 2020 raft_consensus.cc:374] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:40.525394 2020 raft_consensus.cc:729] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3383208265e648d9bea3bbc3222d5e86, State: Initialized, Role: FOLLOWER
I20260619 13:13:40.525429 2017 ts_tablet_manager.cc:1428] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 13:13:40.525597 2020 consensus_queue.cc:260] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:40.526271 2020 ts_tablet_manager.cc:1428] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 13:13:40.732872 2024 raft_consensus.cc:482] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:13:40.733080 2024 raft_consensus.cc:504] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:40.734517 2024 leader_election.cc:288] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0df3db2d34e4367b60d855569f306f4 (127.0.97.1:40843), 1fe03a3a9f034050bd4794ee263556c0 (127.0.97.2:42773)
I20260619 13:13:40.741525 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.735135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55651 (local address 127.0.97.1:40843)
0619 13:13:40.735348 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:40.735355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:40.735778 (+ 423us) server_negotiation.cc:407] Connection header received
0619 13:13:40.736088 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:40.736094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:40.736182 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:40.736283 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:40.736976 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.737926 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.739825 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.740619 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.740744 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.740844 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:40.741129 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 13:13:40.741254 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 13:13:40.741602 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.734791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40843 (local address 127.0.97.3:55651)
0619 13:13:40.735657 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:40.735682 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:40.735883 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:40.736453 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:40.736456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:40.736474 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:40.736802 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.736811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.738049 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.738054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.739668 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.739681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.740775 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.740781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.740946 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.740979 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:13:40.741092 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 13:13:40.741225 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"spinlock_wait_cycles":3456,"thread_start_us":883,"threads_started":1}
I20260619 13:13:40.742491 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d5f0a4ebbe4b539749c97c1f069547" candidate_uuid: "3383208265e648d9bea3bbc3222d5e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0df3db2d34e4367b60d855569f306f4" is_pre_election: true
I20260619 13:13:40.742753 1674 raft_consensus.cc:2454] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3383208265e648d9bea3bbc3222d5e86 in term 0.
I20260619 13:13:40.742836 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.734809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42773 (local address 127.0.97.3:32955)
0619 13:13:40.735879 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:40.735893 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:40.735983 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:40.736425 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:40.736430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:40.736465 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:40.736701 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.736713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.738435 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.738440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.740455 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.740471 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.742056 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.742063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.742243 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.742280 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 13:13:40.742419 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 13:13:40.742543 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":607,"threads_started":1}
I20260619 13:13:40.742863 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.735065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32955 (local address 127.0.97.2:42773)
0619 13:13:40.735290 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:40.735297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:40.735980 (+ 683us) server_negotiation.cc:407] Connection header received
0619 13:13:40.736194 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:40.736200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:40.736279 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:40.736386 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:40.736939 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.738241 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.740633 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.741857 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.741982 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.742072 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:40.742456 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 13:13:40.742586 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 13:13:40.743489 1878 leader_election.cc:302] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3383208265e648d9bea3bbc3222d5e86, b0df3db2d34e4367b60d855569f306f4; no voters:
I20260619 13:13:40.743883 2024 raft_consensus.cc:2781] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:13:40.744001 2024 raft_consensus.cc:482] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:13:40.743840 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d5f0a4ebbe4b539749c97c1f069547" candidate_uuid: "3383208265e648d9bea3bbc3222d5e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe03a3a9f034050bd4794ee263556c0" is_pre_election: true
I20260619 13:13:40.744094 2024 raft_consensus.cc:3037] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:40.744256 1808 raft_consensus.cc:2454] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3383208265e648d9bea3bbc3222d5e86 in term 0.
I20260619 13:13:40.747398 2024 raft_consensus.cc:504] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:40.748163 2024 leader_election.cc:288] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [CANDIDATE]: Term 1 election: Requested vote from peers b0df3db2d34e4367b60d855569f306f4 (127.0.97.1:40843), 1fe03a3a9f034050bd4794ee263556c0 (127.0.97.2:42773)
I20260619 13:13:40.748592 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d5f0a4ebbe4b539749c97c1f069547" candidate_uuid: "3383208265e648d9bea3bbc3222d5e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe03a3a9f034050bd4794ee263556c0"
I20260619 13:13:40.748600 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d5f0a4ebbe4b539749c97c1f069547" candidate_uuid: "3383208265e648d9bea3bbc3222d5e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0df3db2d34e4367b60d855569f306f4"
I20260619 13:13:40.748764 1808 raft_consensus.cc:3037] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:40.748804 1674 raft_consensus.cc:3037] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:40.751042 1808 raft_consensus.cc:2454] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3383208265e648d9bea3bbc3222d5e86 in term 1.
I20260619 13:13:40.751375 1880 leader_election.cc:302] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe03a3a9f034050bd4794ee263556c0, 3383208265e648d9bea3bbc3222d5e86; no voters:
I20260619 13:13:40.751623 2024 raft_consensus.cc:2781] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:40.751598 1674 raft_consensus.cc:2454] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3383208265e648d9bea3bbc3222d5e86 in term 1.
I20260619 13:13:40.754053 2024 raft_consensus.cc:686] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [term 1 LEADER]: Becoming Leader. State: Replica: 3383208265e648d9bea3bbc3222d5e86, State: Running, Role: LEADER
I20260619 13:13:40.754421 2024 consensus_queue.cc:237] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:40.757691 1511 catalog_manager.cc:5495] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3383208265e648d9bea3bbc3222d5e86 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "3383208265e648d9bea3bbc3222d5e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } health_report { overall_health: HEALTHY } } }
Jun 19 13:13:40 dist-test-slave-mkt3 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:40.815790 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.799908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39039 (local address 127.0.0.1:40576)
0619 13:13:40.800562 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:40.800582 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:40.800741 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:40.801168 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:40.801173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:40.802532 (+ 1359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:40.802787 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.802802 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.803866 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.803872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.805311 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:40.805325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.806066 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.806072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:40.806242 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.806957 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:40.806980 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:40.809706 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:40.812768 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:40.812780 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:40.812788 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:40.813359 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:40.813953 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:40.813959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:40.813962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:40.814207 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:40.814908 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:40.814917 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:40.815092 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 13:13:40.815288 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 13:13:40.815527 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":316,"threads_started":1}
I20260619 13:13:40.816453 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:40.799957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40576 (local address 127.0.97.62:39039)
0619 13:13:40.800877 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:40.800881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:40.800900 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:13:40.800940 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:40.800943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:40.800994 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:40.801095 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:40.802967 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.803729 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.805455 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:40.805905 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:40.805997 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:40.809942 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:40.809948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:40.809951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:40.809979 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:40.812584 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:40.813573 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:40.813579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:40.813581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:40.813808 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:40.814332 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:40.814338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:40.814341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:40.814710 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:40.814838 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:40.816073 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0619 13:13:40.816232 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":710,"threads_started":1}
I20260619 13:13:40.821605 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40576:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 13:13:40.822846 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:13:40.829322 1649 tablet_service.cc:1467] Processing CreateTablet for tablet f218be7fffa147d6b3f1082cade0ec6f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d03a43ae1234885b610f375b479ef5a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:13:40.829922 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f218be7fffa147d6b3f1082cade0ec6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:40.829789 1788 tablet_service.cc:1467] Processing CreateTablet for tablet f218be7fffa147d6b3f1082cade0ec6f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d03a43ae1234885b610f375b479ef5a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:13:40.830168 1927 tablet_service.cc:1467] Processing CreateTablet for tablet f218be7fffa147d6b3f1082cade0ec6f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d03a43ae1234885b610f375b479ef5a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:13:40.830340 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f218be7fffa147d6b3f1082cade0ec6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:40.830632 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f218be7fffa147d6b3f1082cade0ec6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:40.837229 2017 tablet_bootstrap.cc:492] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4: Bootstrap starting.
I20260619 13:13:40.837958 2020 tablet_bootstrap.cc:492] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86: Bootstrap starting.
I20260619 13:13:40.838703 2017 tablet_bootstrap.cc:654] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:40.839224 2020 tablet_bootstrap.cc:654] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:40.841207 2020 tablet_bootstrap.cc:492] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86: No bootstrap required, opened a new log
I20260619 13:13:40.841346 2020 ts_tablet_manager.cc:1397] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 13:13:40.841956 2020 raft_consensus.cc:348] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:40.842105 2016 tablet_bootstrap.cc:492] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0: Bootstrap starting.
I20260619 13:13:40.842263 2020 raft_consensus.cc:374] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:40.842398 2020 raft_consensus.cc:729] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3383208265e648d9bea3bbc3222d5e86, State: Initialized, Role: FOLLOWER
I20260619 13:13:40.842687 2020 consensus_queue.cc:260] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:40.843015 2024 tablet_replica.cc:439] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } }
I20260619 13:13:40.843266 2024 tablet_replica.cc:442] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:13:40.843858 2016 tablet_bootstrap.cc:654] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:40.843905 2020 ts_tablet_manager.cc:1428] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 13:13:40.845690 2017 tablet_bootstrap.cc:492] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4: No bootstrap required, opened a new log
I20260619 13:13:40.846340 2016 tablet_bootstrap.cc:492] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0: No bootstrap required, opened a new log
I20260619 13:13:40.846495 2016 ts_tablet_manager.cc:1397] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 13:13:40.847162 2016 raft_consensus.cc:348] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:40.847400 2016 raft_consensus.cc:374] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:40.847473 2016 raft_consensus.cc:729] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe03a3a9f034050bd4794ee263556c0, State: Initialized, Role: FOLLOWER
I20260619 13:13:40.847690 2016 consensus_queue.cc:260] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:40.848241 2016 ts_tablet_manager.cc:1428] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 13:13:40.848124 2035 tablet_replica.cc:439] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } }
I20260619 13:13:40.848531 2035 tablet_replica.cc:442] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:13:40.848724 2017 ts_tablet_manager.cc:1397] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20260619 13:13:40.849404 2017 raft_consensus.cc:348] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:40.849677 2017 raft_consensus.cc:374] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:40.849786 2017 raft_consensus.cc:729] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0df3db2d34e4367b60d855569f306f4, State: Initialized, Role: FOLLOWER
I20260619 13:13:40.850061 2017 consensus_queue.cc:260] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:40.850508 2022 tablet_replica.cc:439] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } }
I20260619 13:13:40.850800 2022 tablet_replica.cc:442] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:13:40.851581 2017 ts_tablet_manager.cc:1428] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260619 13:13:41.055344 2035 raft_consensus.cc:482] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:13:41.055510 2035 raft_consensus.cc:504] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:41.056115 2035 leader_election.cc:288] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0df3db2d34e4367b60d855569f306f4 (127.0.97.1:40843), 3383208265e648d9bea3bbc3222d5e86 (127.0.97.3:43905)
I20260619 13:13:41.056607 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f218be7fffa147d6b3f1082cade0ec6f" candidate_uuid: "1fe03a3a9f034050bd4794ee263556c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0df3db2d34e4367b60d855569f306f4" is_pre_election: true
I20260619 13:13:41.056712 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f218be7fffa147d6b3f1082cade0ec6f" candidate_uuid: "1fe03a3a9f034050bd4794ee263556c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3383208265e648d9bea3bbc3222d5e86" is_pre_election: true
I20260619 13:13:41.056879 1674 raft_consensus.cc:2454] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fe03a3a9f034050bd4794ee263556c0 in term 0.
I20260619 13:13:41.056916 1948 raft_consensus.cc:2454] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fe03a3a9f034050bd4794ee263556c0 in term 0.
I20260619 13:13:41.057463 1739 leader_election.cc:302] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe03a3a9f034050bd4794ee263556c0, b0df3db2d34e4367b60d855569f306f4; no voters:
I20260619 13:13:41.057828 2035 raft_consensus.cc:2781] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:13:41.057979 2035 raft_consensus.cc:482] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:13:41.058079 2035 raft_consensus.cc:3037] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:41.060851 2035 raft_consensus.cc:504] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:41.061475 2035 leader_election.cc:288] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [CANDIDATE]: Term 1 election: Requested vote from peers b0df3db2d34e4367b60d855569f306f4 (127.0.97.1:40843), 3383208265e648d9bea3bbc3222d5e86 (127.0.97.3:43905)
I20260619 13:13:41.062033 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f218be7fffa147d6b3f1082cade0ec6f" candidate_uuid: "1fe03a3a9f034050bd4794ee263556c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0df3db2d34e4367b60d855569f306f4"
I20260619 13:13:41.062058 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f218be7fffa147d6b3f1082cade0ec6f" candidate_uuid: "1fe03a3a9f034050bd4794ee263556c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3383208265e648d9bea3bbc3222d5e86"
I20260619 13:13:41.062239 1948 raft_consensus.cc:3037] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:41.062266 1674 raft_consensus.cc:3037] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:41.064137 1948 raft_consensus.cc:2454] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe03a3a9f034050bd4794ee263556c0 in term 1.
I20260619 13:13:41.064319 1674 raft_consensus.cc:2454] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe03a3a9f034050bd4794ee263556c0 in term 1.
I20260619 13:13:41.064494 1739 leader_election.cc:302] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fe03a3a9f034050bd4794ee263556c0, 3383208265e648d9bea3bbc3222d5e86; no voters:
I20260619 13:13:41.064766 2035 raft_consensus.cc:2781] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:41.065172 2035 raft_consensus.cc:686] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [term 1 LEADER]: Becoming Leader. State: Replica: 1fe03a3a9f034050bd4794ee263556c0, State: Running, Role: LEADER
I20260619 13:13:41.065495 2035 consensus_queue.cc:237] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } }
I20260619 13:13:41.066992 2060 tablet_replica.cc:439] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0: TxnStatusTablet state changed. Reason: New leader 1fe03a3a9f034050bd4794ee263556c0. Latest consensus state: current_term: 1 leader_uuid: "1fe03a3a9f034050bd4794ee263556c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } }
I20260619 13:13:41.067240 2060 tablet_replica.cc:442] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:13:41.067728 2062 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 13:13:41.067956 2062 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 13:13:41.068255 1512 catalog_manager.cc:5495] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fe03a3a9f034050bd4794ee263556c0 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1fe03a3a9f034050bd4794ee263556c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } health_report { overall_health: UNKNOWN } } }
I20260619 13:13:41.090656 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:41.084914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.0.97.2:42773)
0619 13:13:41.085150 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:41.085158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:41.085700 (+ 542us) server_negotiation.cc:407] Connection header received
0619 13:13:41.085960 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:41.085963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:41.086024 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:41.086114 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:13:41.086460 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.087201 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.089012 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.089414 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.089501 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:41.090044 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:13:41.090149 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:13:41.090225 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:41.090380 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 13:13:41.090448 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:13:41.090812 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:41.085372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42773 (local address 127.0.0.1:48212)
0619 13:13:41.085583 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:41.085602 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:41.085772 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:41.086143 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:41.086148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:41.086160 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:13:41.086335 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:41.086346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.087319 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.087324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:41.088874 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:41.088888 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.089549 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.089555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:41.089776 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:41.089867 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:13:41.090248 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:13:41.090267 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 13:13:41.090356 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 13:13:41.090507 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 13:13:41.094492 1948 raft_consensus.cc:1261] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86 [term 1 FOLLOWER]: Refusing update from remote peer 1fe03a3a9f034050bd4794ee263556c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:13:41.094576 1674 raft_consensus.cc:1261] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4 [term 1 FOLLOWER]: Refusing update from remote peer 1fe03a3a9f034050bd4794ee263556c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:13:41.095108 2060 consensus_queue.cc:1035] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:13:41.095393 2035 consensus_queue.cc:1035] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:13:41.111357 2024 tablet_replica.cc:439] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86: TxnStatusTablet state changed. Reason: New leader 1fe03a3a9f034050bd4794ee263556c0. Latest consensus state: current_term: 1 leader_uuid: "1fe03a3a9f034050bd4794ee263556c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } }
I20260619 13:13:41.111675 2024 tablet_replica.cc:442] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:13:41.122156 2060 tablet_replica.cc:439] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1fe03a3a9f034050bd4794ee263556c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } }
I20260619 13:13:41.122442 2060 tablet_replica.cc:442] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:13:41.125387 2067 mvcc.cc:205] Tried to move back new op lower bound from 7298559267197612032 to 7298559267086589952. Current Snapshot: MvccSnapshot[applied={T|T < 7298559267197612032}]
I20260619 13:13:41.132197 2022 tablet_replica.cc:439] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4: TxnStatusTablet state changed. Reason: New leader 1fe03a3a9f034050bd4794ee263556c0. Latest consensus state: current_term: 1 leader_uuid: "1fe03a3a9f034050bd4794ee263556c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } }
I20260619 13:13:41.132498 2022 tablet_replica.cc:442] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:13:41.135221 2035 tablet_replica.cc:439] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1fe03a3a9f034050bd4794ee263556c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } }
I20260619 13:13:41.135494 2035 tablet_replica.cc:442] T f218be7fffa147d6b3f1082cade0ec6f P 1fe03a3a9f034050bd4794ee263556c0: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:13:41.137537 2066 mvcc.cc:205] Tried to move back new op lower bound from 7298559267197612032 to 7298559267086589952. Current Snapshot: MvccSnapshot[applied={T|T < 7298559267197612032}]
I20260619 13:13:41.141989 2024 tablet_replica.cc:439] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1fe03a3a9f034050bd4794ee263556c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } }
I20260619 13:13:41.142280 2024 tablet_replica.cc:442] T f218be7fffa147d6b3f1082cade0ec6f P 3383208265e648d9bea3bbc3222d5e86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:13:41.150797 2022 tablet_replica.cc:439] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1fe03a3a9f034050bd4794ee263556c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 } } peers { permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 } } peers { permanent_uuid: "3383208265e648d9bea3bbc3222d5e86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43905 } } }
I20260619 13:13:41.151051 2022 tablet_replica.cc:442] T f218be7fffa147d6b3f1082cade0ec6f P b0df3db2d34e4367b60d855569f306f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:13:41.158423 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:41.144239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.0.97.3:43905)
0619 13:13:41.150879 (+ 6640us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:41.150885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:41.151490 (+ 605us) server_negotiation.cc:407] Connection header received
0619 13:13:41.151700 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:41.151705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:41.151803 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:41.153327 (+ 1524us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:13:41.153484 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.154302 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.156202 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.156586 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.156681 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:41.157231 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:13:41.157777 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:13:41.157949 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:41.158036 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 13:13:41.158109 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6517,"thread_start_us":6398,"threads_started":1}
I20260619 13:13:41.160854 2024 consensus_queue.cc:1035] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0df3db2d34e4367b60d855569f306f4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:13:41.167088 2024 consensus_queue.cc:1035] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe03a3a9f034050bd4794ee263556c0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:13:41.178845 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:41.171736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47847 (local address 127.0.97.2:42773)
0619 13:13:41.171934 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:41.171940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:41.172966 (+ 1026us) server_negotiation.cc:407] Connection header received
0619 13:13:41.173139 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:41.173144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:41.173219 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:41.173293 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:13:41.173704 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.174775 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.176658 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.177351 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.177461 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:41.178125 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:13:41.178272 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:13:41.178348 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:41.178526 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 13:13:41.178592 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:13:41.179047 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:41.171668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42773 (local address 127.0.97.3:47847)
0619 13:13:41.172868 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:41.172887 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:41.173016 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:41.173318 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:41.173322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:41.173333 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:13:41.173562 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:41.173576 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.174950 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.174955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:41.176517 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:41.176531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.177655 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.177662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:41.177869 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:41.177938 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:13:41.178394 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:13:41.178416 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:13:41.178493 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 13:13:41.178636 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"thread_start_us":929,"threads_started":1}
I20260619 13:13:41.216414 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:41.209197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43549 (local address 127.0.97.3:43905)
0619 13:13:41.209389 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:41.209396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:41.209878 (+ 482us) server_negotiation.cc:407] Connection header received
0619 13:13:41.210098 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:41.210103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:41.210182 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:41.210263 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:13:41.210990 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.212095 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.213890 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.215042 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.215158 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:41.215740 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:13:41.215833 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:13:41.215897 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:41.216016 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 13:13:41.216096 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:13:41.216540 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:41.209153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43905 (local address 127.0.97.2:43549)
0619 13:13:41.209750 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:41.209773 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:41.209943 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:41.210286 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:41.210290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:41.210302 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:13:41.210638 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:41.210652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.212234 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.212237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:41.213542 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:41.213557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.215288 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.215294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:41.215481 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:41.215558 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:13:41.215921 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:13:41.215932 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 13:13:41.216007 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 13:13:41.216147 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":248,"threads_started":1}
I20260619 13:13:41.252506 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:13:41.252772 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:13:41.256031 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a07e43a7aee4543936faee342ddaef3: Sending DeleteTablet for 3 replicas of tablet e2d5f0a4ebbe4b539749c97c1f069547
I20260619 13:13:41.256749 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d5f0a4ebbe4b539749c97c1f069547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36228
I20260619 13:13:41.257093 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d5f0a4ebbe4b539749c97c1f069547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57496
I20260619 13:13:41.257644 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d5f0a4ebbe4b539749c97c1f069547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48204
I20260619 13:13:41.269181 2088 tablet_replica.cc:330] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4: stopping tablet replica
I20260619 13:13:41.269483 2088 raft_consensus.cc:2229] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:13:41.269757 2088 raft_consensus.cc:2258] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:41.270867 2088 ts_tablet_manager.cc:1905] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:41.278450 2088 ts_tablet_manager.cc:1918] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:13:41.278687 2088 log.cc:1196] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/e2d5f0a4ebbe4b539749c97c1f069547
I20260619 13:13:41.279161 2088 ts_tablet_manager.cc:1939] T e2d5f0a4ebbe4b539749c97c1f069547 P b0df3db2d34e4367b60d855569f306f4: Deleting consensus metadata
I20260619 13:13:41.280287 1496 catalog_manager.cc:4841] TS b0df3db2d34e4367b60d855569f306f4 (127.0.97.1:40843): tablet e2d5f0a4ebbe4b539749c97c1f069547 (table test-table [id=5d8d0777e86f4c2d814eb36d68136ef1]) successfully deleted
I20260619 13:13:41.282328 2091 tablet_replica.cc:330] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0: stopping tablet replica
I20260619 13:13:41.282584 2091 raft_consensus.cc:2229] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:13:41.283905 2091 raft_consensus.cc:2258] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:41.285038 2091 ts_tablet_manager.cc:1905] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:41.286036 2089 tablet_replica.cc:330] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86: stopping tablet replica
I20260619 13:13:41.287367 2089 raft_consensus.cc:2229] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:13:41.287945 2089 raft_consensus.cc:2258] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:41.294727 2089 ts_tablet_manager.cc:1905] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:41.302202 2091 ts_tablet_manager.cc:1918] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:13:41.302345 2091 log.cc:1196] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/e2d5f0a4ebbe4b539749c97c1f069547
I20260619 13:13:41.302454 2089 ts_tablet_manager.cc:1918] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:13:41.302618 2089 log.cc:1196] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/e2d5f0a4ebbe4b539749c97c1f069547
I20260619 13:13:41.303081 2089 ts_tablet_manager.cc:1939] T e2d5f0a4ebbe4b539749c97c1f069547 P 3383208265e648d9bea3bbc3222d5e86: Deleting consensus metadata
I20260619 13:13:41.304250 1496 catalog_manager.cc:4841] TS 3383208265e648d9bea3bbc3222d5e86 (127.0.97.3:43905): tablet e2d5f0a4ebbe4b539749c97c1f069547 (table test-table [id=5d8d0777e86f4c2d814eb36d68136ef1]) successfully deleted
I20260619 13:13:41.306010 2091 ts_tablet_manager.cc:1939] T e2d5f0a4ebbe4b539749c97c1f069547 P 1fe03a3a9f034050bd4794ee263556c0: Deleting consensus metadata
I20260619 13:13:41.307054 1498 catalog_manager.cc:4841] TS 1fe03a3a9f034050bd4794ee263556c0 (127.0.97.2:42773): tablet e2d5f0a4ebbe4b539749c97c1f069547 (table test-table [id=5d8d0777e86f4c2d814eb36d68136ef1]) successfully deleted
Jun 19 13:13:41 dist-test-slave-mkt3 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:13:41.318624 2092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:41.283820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.0.97.1:40843)
0619 13:13:41.296367 (+ 12547us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:41.296374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:41.298690 (+ 2316us) server_negotiation.cc:407] Connection header received
0619 13:13:41.298756 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:41.298762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:41.298839 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:41.298933 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:41.302813 (+ 3880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.303935 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.305784 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.306492 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.306630 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:41.311424 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:41.311441 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:41.311462 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:41.311506 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:41.315219 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:41.316006 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:41.316016 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:41.316020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:41.316174 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:41.316555 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:41.316561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:41.316564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:41.316972 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:41.317249 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:41.317562 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 13:13:41.317763 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12400,"thread_start_us":12554,"threads_started":1}
W20260619 13:13:41.319388 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36244
I20260619 13:13:41.346891 2095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:41.322332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.0.97.62:39039)
0619 13:13:41.330285 (+ 7953us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:41.330292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:41.330313 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:13:41.330371 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:41.330377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:41.330457 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:41.330563 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:41.332264 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.333438 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.335800 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.336483 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.336613 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:41.339491 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:41.339498 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:41.339504 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:41.339559 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:41.343544 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:41.344353 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:41.344359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:41.344363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:41.344539 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:41.344966 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:41.344972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:41.344975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:41.345317 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:41.345549 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:41.346136 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 13:13:41.346357 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7831,"thread_start_us":7706,"threads_started":1}
W20260619 13:13:41.347894 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40580
I20260619 13:13:41.350162 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 1583
I20260619 13:13:41.368404 2092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:41.352382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.0.97.1:40843)
0619 13:13:41.352564 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:41.352569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:41.354354 (+ 1785us) server_negotiation.cc:407] Connection header received
0619 13:13:41.354509 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:41.354515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:41.354594 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:41.354697 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:41.356056 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.357164 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.358934 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.359552 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.359682 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:41.362464 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:41.362470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:41.362475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:41.362513 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:41.365755 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:41.366500 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:41.366505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:41.366509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:41.366663 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:41.367037 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:41.367041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:41.367045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:41.367338 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:41.367573 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:41.367861 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 13:13:41.368068 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 13:13:41.369238 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36258
W20260619 13:13:41.369876 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 1583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:13:41.370055 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 1583
I20260619 13:13:41.389966 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 1723
Jun 19 13:13:41 dist-test-slave-mkt3 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:13:41.409224 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:41.390923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48214 (local address 127.0.97.2:42773)
0619 13:13:41.391101 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:41.391111 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:41.391130 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:13:41.391239 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:41.391248 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:41.391323 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:41.391421 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:41.393016 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.394293 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.396069 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.396722 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.396875 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:41.401941 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:41.401959 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:41.401979 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:41.402027 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:41.405999 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:41.406797 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:41.406806 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:41.406810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:41.406969 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:41.407374 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:41.407379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:41.407383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:41.407848 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:41.408156 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:41.408428 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 13:13:41.408677 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 13:13:41.409914 1853 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48214
W20260619 13:13:41.410642 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 1723: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:13:41.410815 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 1723
I20260619 13:13:41.430441 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 1862
Jun 19 13:13:41 dist-test-slave-mkt3 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:13:41.445572 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:41.431189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.0.97.3:43905)
0619 13:13:41.431372 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:41.431378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:41.431398 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:13:41.431513 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:41.431518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:41.431622 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:41.431737 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:41.432825 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.433984 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.435213 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:41.435903 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:41.436022 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:41.440109 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:41.440125 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:41.440141 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:41.440171 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:41.442977 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:41.443689 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:41.443696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:41.443698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:41.443817 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:41.444174 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:41.444177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:41.444179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:41.444483 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:41.444720 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:41.445074 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 13:13:41.445228 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 13:13:41.446285 1992 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57516
W20260619 13:13:41.446919 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 1862: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:13:41.447063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 1862
I20260619 13:13:41.461078 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 1480
I20260619 13:13:41.462080 1552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:13:41.570348 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 1480
2026-06-19T13:13:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4618 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:13:41 dist-test-slave-mkt3 krb5kdc[2101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:13:41 dist-test-slave-mkt3 krb5kdc[2101](info): set up 2 sockets
Jun 19 13:13:41 dist-test-slave-mkt3 krb5kdc[2101](info): commencing operation
krb5kdc: starting...
W20260619 13:13:42.561668 388 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.011s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:13:42 dist-test-slave-mkt3 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:13:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:13:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:13:42.719197 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39397
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:43.017194 2114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:43.017455 2114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:43.017567 2114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:43.024580 2114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:13:43.024675 2114 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 13:13:43.024726 2114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:43.024811 2114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:43.024863 2114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:13:43.024942 2114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:13:43.034444 2114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39397
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:43.035586 2114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:43.036859 2114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:43.047681 2119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:43.047768 2120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:43.047712 2122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:43.048568 2114 server_base.cc:1034] running on GCE node
I20260619 13:13:43.049261 2114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:43.050585 2114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:43.051812 2114 hybrid_clock.cc:648] HybridClock initialized: now 1781874823051739 us; error 49 us; skew 500 ppm
I20260619 13:13:43.052147 2114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:43 dist-test-slave-mkt3 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:43.059674 2114 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:13:43.061064 2114 webserver.cc:460] Webserver started at http://127.0.97.62:35851/ using document root <none> and password file <none>
I20260619 13:13:43.061758 2114 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:43.061872 2114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:43.062165 2114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:43.064888 2114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "e42fb3e416e54d859259ae7196d677de"
format_stamp: "Formatted at 2026-06-19 13:13:43 on dist-test-slave-mkt3"
server_key: "8c9815011554c69c0298da416a145c2c"
server_key_iv: "c42b43e404aed86f553cec2605e83d29"
server_key_version: "encryptionkey@0"
I20260619 13:13:43.065590 2114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "e42fb3e416e54d859259ae7196d677de"
format_stamp: "Formatted at 2026-06-19 13:13:43 on dist-test-slave-mkt3"
server_key: "8c9815011554c69c0298da416a145c2c"
server_key_iv: "c42b43e404aed86f553cec2605e83d29"
server_key_version: "encryptionkey@0"
I20260619 13:13:43.070338 2114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:13:43.073552 2129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:43.074780 2114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:13:43.074923 2114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "e42fb3e416e54d859259ae7196d677de"
format_stamp: "Formatted at 2026-06-19 13:13:43 on dist-test-slave-mkt3"
server_key: "8c9815011554c69c0298da416a145c2c"
server_key_iv: "c42b43e404aed86f553cec2605e83d29"
server_key_version: "encryptionkey@0"
I20260619 13:13:43.075124 2114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:43.095018 2114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:43.098930 2114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:43.099156 2114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:43.117503 2114 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39397
I20260619 13:13:43.117522 2190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39397 every 8 connection(s)
I20260619 13:13:43.118768 2114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:13:43.122185 2191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:43.122807 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 2114
I20260619 13:13:43.122987 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:13:43.123557 388 external_mini_cluster.cc:1428] Setting key a6b23f2b3f7eecb628b2f06b403e7606
I20260619 13:13:43.135169 2191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de: Bootstrap starting.
Jun 19 13:13:43 dist-test-slave-mkt3 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:43.138278 2191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:43.139526 2191 log.cc:824] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:43.142591 2191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de: No bootstrap required, opened a new log
I20260619 13:13:43.146057 2194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:43.125433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.0.97.62:39397)
0619 13:13:43.126212 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:43.126238 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:43.126292 (+ 54us) server_negotiation.cc:407] Connection header received
0619 13:13:43.128067 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:43.128093 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:43.128593 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:43.129016 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:43.130252 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:43.132090 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:43.133318 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:43.134313 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:43.134478 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:43.138330 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:43.138340 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:43.138359 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:43.138401 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:43.141878 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:43.142497 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:43.142505 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:43.142513 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:43.142741 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:43.143102 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:43.143106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:43.143109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:43.143733 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:43.144007 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:43.144352 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 13:13:43.144759 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":326,"threads_started":1}
I20260619 13:13:43.151293 2191 raft_consensus.cc:348] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42fb3e416e54d859259ae7196d677de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39397 } }
I20260619 13:13:43.151589 2191 raft_consensus.cc:374] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:43.151659 2191 raft_consensus.cc:729] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e42fb3e416e54d859259ae7196d677de, State: Initialized, Role: FOLLOWER
I20260619 13:13:43.152118 2191 consensus_queue.cc:260] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42fb3e416e54d859259ae7196d677de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39397 } }
I20260619 13:13:43.152299 2191 raft_consensus.cc:388] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:13:43.152383 2191 raft_consensus.cc:482] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:13:43.152496 2191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:43.154461 2191 raft_consensus.cc:504] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42fb3e416e54d859259ae7196d677de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39397 } }
I20260619 13:13:43.154891 2191 leader_election.cc:302] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e42fb3e416e54d859259ae7196d677de; no voters:
I20260619 13:13:43.155418 2191 leader_election.cc:288] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:13:43.155506 2196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:43.156399 2196 raft_consensus.cc:686] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [term 1 LEADER]: Becoming Leader. State: Replica: e42fb3e416e54d859259ae7196d677de, State: Running, Role: LEADER
I20260619 13:13:43.156859 2196 consensus_queue.cc:237] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42fb3e416e54d859259ae7196d677de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39397 } }
I20260619 13:13:43.157347 2191 sys_catalog.cc:564] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:13:43.159356 2197 sys_catalog.cc:455] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e42fb3e416e54d859259ae7196d677de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42fb3e416e54d859259ae7196d677de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39397 } } }
I20260619 13:13:43.159654 2197 sys_catalog.cc:458] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:43.160404 2203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:13:43.160239 2198 sys_catalog.cc:455] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [sys.catalog]: SysCatalogTable state changed. Reason: New leader e42fb3e416e54d859259ae7196d677de. Latest consensus state: current_term: 1 leader_uuid: "e42fb3e416e54d859259ae7196d677de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42fb3e416e54d859259ae7196d677de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39397 } } }
I20260619 13:13:43.161517 2198 sys_catalog.cc:458] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:43.165484 2203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:13:43.175134 2203 catalog_manager.cc:1347] Generated new cluster ID: 7085d8ed35114442b2bc2b597dd95428
I20260619 13:13:43.175253 2203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:13:43.200140 2203 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:13:43.201521 2203 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:13:43.220993 2203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de: Generated new TSK 0
I20260619 13:13:43.221976 2203 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:13:43.259490 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39397
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:43.556967 2217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:43.557240 2217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:43.557374 2217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:43.564347 2217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:43.564441 2217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:43.564561 2217 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 13:13:43.564622 2217 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:13:43.564719 2217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:13:43.574091 2217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:43.575233 2217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:43.576444 2217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:43.588423 2223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:43.588524 2222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:43.588832 2225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:43.589193 2217 server_base.cc:1034] running on GCE node
I20260619 13:13:43.589711 2217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:43.590571 2217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:43.591758 2217 hybrid_clock.cc:648] HybridClock initialized: now 1781874823591713 us; error 39 us; skew 500 ppm
I20260619 13:13:43.592026 2217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:43 dist-test-slave-mkt3 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:43.594786 2217 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:13:43.596076 2217 webserver.cc:460] Webserver started at http://127.0.97.1:36201/ using document root <none> and password file <none>
I20260619 13:13:43.596663 2217 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:43.596771 2217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:43.597038 2217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:43.599681 2217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "c3a8968dc803419d9eb8ca8e0cf2f052"
format_stamp: "Formatted at 2026-06-19 13:13:43 on dist-test-slave-mkt3"
server_key: "131456381cc8f45ddd4ef66798dcdbd3"
server_key_iv: "163628dd373545c4526280df91331383"
server_key_version: "encryptionkey@0"
I20260619 13:13:43.600399 2217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c3a8968dc803419d9eb8ca8e0cf2f052"
format_stamp: "Formatted at 2026-06-19 13:13:43 on dist-test-slave-mkt3"
server_key: "131456381cc8f45ddd4ef66798dcdbd3"
server_key_iv: "163628dd373545c4526280df91331383"
server_key_version: "encryptionkey@0"
I20260619 13:13:43.605011 2217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 13:13:43.608106 2232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:43.609283 2217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 13:13:43.609406 2217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "c3a8968dc803419d9eb8ca8e0cf2f052"
format_stamp: "Formatted at 2026-06-19 13:13:43 on dist-test-slave-mkt3"
server_key: "131456381cc8f45ddd4ef66798dcdbd3"
server_key_iv: "163628dd373545c4526280df91331383"
server_key_version: "encryptionkey@0"
I20260619 13:13:43.609568 2217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:43.642860 2217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:43.646739 2217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:43.646974 2217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:43.649374 2217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:43.649458 2217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:43.649554 2217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:43.649627 2217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:13:43 dist-test-slave-mkt3 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:43.669828 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:43.650178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47513 (local address 127.0.97.62:39397)
0619 13:13:43.652260 (+ 2082us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:43.652266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:43.652288 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:13:43.654061 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:43.654069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:43.654160 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:43.654286 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:43.656257 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:43.657345 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:43.659497 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:43.660153 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:43.660286 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:43.664116 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:43.664124 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:43.664128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:43.664167 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:43.667154 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:43.667796 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:43.667802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:43.667806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:43.667916 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:43.668298 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:43.668303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:43.668307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:43.668552 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:43.668681 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:43.669295 (+ 614us) server_negotiation.cc:299] Negotiation successful
0619 13:13:43.669492 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1971,"thread_start_us":1855,"threads_started":1}
I20260619 13:13:43.670759 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:43.650699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39397 (local address 127.0.97.1:47513)
0619 13:13:43.651507 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:43.651554 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:43.653710 (+ 2156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:43.654559 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:43.654571 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:43.655291 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:43.656056 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:43.656081 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:43.657597 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:43.657602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:43.659360 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:43.659372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:43.660385 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:43.660390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:43.660574 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:43.661179 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:43.661206 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:43.663880 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:43.667280 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:43.667289 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:43.667305 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:43.667685 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:43.668006 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:43.668011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:43.668015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:43.668195 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:43.668663 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:43.668670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:43.668978 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 13:13:43.669195 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 13:13:43.669664 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":374,"threads_started":1}
I20260619 13:13:43.695443 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:43.680356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39397 (local address 127.0.97.1:41037)
0619 13:13:43.680981 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:43.681016 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:43.681206 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:43.681713 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:43.681718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:43.682333 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:43.682730 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:43.682743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:43.684114 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:43.684120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:43.685919 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:43.685934 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:43.686902 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:43.686913 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:43.687153 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:43.687857 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:43.687887 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:43.689060 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:43.692254 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:43.692259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:43.692263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:43.692872 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:43.693282 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:43.693287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:43.693291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:43.693454 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:43.694052 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:43.694057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:43.694286 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 13:13:43.694454 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 13:13:43.694810 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":339,"threads_started":1}
I20260619 13:13:43.695967 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:43.680255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41037 (local address 127.0.97.62:39397)
0619 13:13:43.680427 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:43.680433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:43.681097 (+ 664us) server_negotiation.cc:407] Connection header received
0619 13:13:43.681371 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:43.681377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:43.681448 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:43.681514 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:43.682902 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:43.683941 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:43.686067 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:43.686691 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:43.686814 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:43.689243 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:43.689249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:43.689254 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:43.689292 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:43.692110 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:43.692990 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:43.692996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:43.692999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:43.693156 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:43.693578 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:43.693583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:43.693585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:43.693928 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:43.694085 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:43.695407 (+ 1322us) server_negotiation.cc:299] Negotiation successful
0619 13:13:43.695645 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:13:43.698717 2217 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37237
I20260619 13:13:43.698781 2352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37237 every 8 connection(s)
I20260619 13:13:43.699965 2217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:13:43.701644 2353 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39397
I20260619 13:13:43.701866 2353 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:43.702427 2353 heartbeater.cc:511] Master 127.0.97.62:39397 requested a full tablet report, sending...
I20260619 13:13:43.704098 2146 ts_manager.cc:194] Registered new tserver with Master: c3a8968dc803419d9eb8ca8e0cf2f052 (127.0.97.1:37237)
I20260619 13:13:43.704171 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 2217
I20260619 13:13:43.704335 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:13:43.704903 388 external_mini_cluster.cc:1428] Setting key 393e7c1236e2de77f764dc4db2f6f1f9
I20260619 13:13:43.705542 2146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47513
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:13:43.738373 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39397
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:44.036918 2357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:44.037177 2357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:44.037305 2357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:44.044389 2357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:44.044518 2357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:44.044656 2357 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 13:13:44.044731 2357 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:13:44.044804 2357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:13:44.054150 2357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:44.055275 2357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:44.056509 2357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:44.068677 2363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:44.068747 2362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:44.068797 2365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:44.069324 2357 server_base.cc:1034] running on GCE node
I20260619 13:13:44.069905 2357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:44.070760 2357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:44.071947 2357 hybrid_clock.cc:648] HybridClock initialized: now 1781874824071905 us; error 32 us; skew 500 ppm
I20260619 13:13:44.072242 2357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:44 dist-test-slave-mkt3 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:44.075060 2357 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:13:44.076362 2357 webserver.cc:460] Webserver started at http://127.0.97.2:43147/ using document root <none> and password file <none>
I20260619 13:13:44.076985 2357 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:44.077080 2357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:44.077316 2357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:44.080055 2357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "2794dbae640c42cca0a0703181f52f09"
format_stamp: "Formatted at 2026-06-19 13:13:44 on dist-test-slave-mkt3"
server_key: "8b6ea122a7b83ecdd609a14559202c69"
server_key_iv: "0d40ad5125044f0b8ff5799d29df234d"
server_key_version: "encryptionkey@0"
I20260619 13:13:44.080755 2357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2794dbae640c42cca0a0703181f52f09"
format_stamp: "Formatted at 2026-06-19 13:13:44 on dist-test-slave-mkt3"
server_key: "8b6ea122a7b83ecdd609a14559202c69"
server_key_iv: "0d40ad5125044f0b8ff5799d29df234d"
server_key_version: "encryptionkey@0"
I20260619 13:13:44.085455 2357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 13:13:44.088657 2372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:44.089946 2357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 13:13:44.090101 2357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "2794dbae640c42cca0a0703181f52f09"
format_stamp: "Formatted at 2026-06-19 13:13:44 on dist-test-slave-mkt3"
server_key: "8b6ea122a7b83ecdd609a14559202c69"
server_key_iv: "0d40ad5125044f0b8ff5799d29df234d"
server_key_version: "encryptionkey@0"
I20260619 13:13:44.090303 2357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:44.141599 2357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:44.145370 2357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:44.145589 2357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:44.148015 2357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:44.148101 2357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:44.148211 2357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:44.148291 2357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:13:44 dist-test-slave-mkt3 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:44.171214 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.148804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45361 (local address 127.0.97.62:39397)
0619 13:13:44.148954 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:44.148958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:44.150434 (+ 1476us) server_negotiation.cc:407] Connection header received
0619 13:13:44.152443 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:44.152449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:44.152529 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:44.152625 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:44.155068 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.155976 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.158943 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.159491 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.159609 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.164801 (+ 5192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:44.164808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:44.164813 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:44.164855 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:44.167691 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:44.168665 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:44.168670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:44.168673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:44.168824 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:44.169308 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:44.169314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:44.169317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:44.169669 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:44.169820 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:44.170682 (+ 862us) server_negotiation.cc:299] Negotiation successful
0619 13:13:44.170886 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 13:13:44.172304 2381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.149467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39397 (local address 127.0.97.2:45361)
0619 13:13:44.150284 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:44.150325 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:44.152101 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:44.152992 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:44.153004 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:44.153930 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:44.154866 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.154893 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.156129 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.156135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.158774 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.158789 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.159912 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.159919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.160252 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.161026 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:44.161061 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:44.164468 (+ 3407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:44.167823 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:44.167834 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:44.167856 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:44.168525 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:44.168926 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:44.168935 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:44.168939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:44.169173 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:44.169834 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:44.169863 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:44.170253 (+ 390us) client_negotiation.cc:769] Sending connection context
0619 13:13:44.170561 (+ 308us) client_negotiation.cc:240] Negotiation successful
0619 13:13:44.171058 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":393,"threads_started":1}
I20260619 13:13:44.190825 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.177520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39397 (local address 127.0.97.2:51751)
0619 13:13:44.178175 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:44.178199 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:44.178394 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:44.178713 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:44.178717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:44.179266 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:44.179624 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.179635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.180682 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.180687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.182431 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.182447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.183268 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.183275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.183460 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.184102 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:44.184124 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:44.185156 (+ 1032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:44.188204 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:44.188209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:44.188213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:44.188778 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:44.189143 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:44.189149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:44.189153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:44.189318 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:44.190007 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:44.190015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:44.190190 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 13:13:44.190298 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 13:13:44.190531 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":342,"threads_started":1}
I20260619 13:13:44.190964 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.177501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51751 (local address 127.0.97.62:39397)
0619 13:13:44.177673 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:44.177679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:44.178290 (+ 611us) server_negotiation.cc:407] Connection header received
0619 13:13:44.178529 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:44.178532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:44.178588 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:44.178692 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:44.179760 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.180547 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.182600 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.183089 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.183182 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.185345 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:44.185350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:44.185353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:44.185381 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:44.188088 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:44.188898 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:44.188903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:44.188907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:44.189037 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:44.189436 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:44.189441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:44.189444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:44.189890 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:44.190032 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:44.190412 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 13:13:44.190630 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:13:44.200543 2357 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45659
I20260619 13:13:44.200603 2491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45659 every 8 connection(s)
I20260619 13:13:44.201706 2357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:13:44.203418 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 2357
I20260619 13:13:44.203469 2492 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39397
I20260619 13:13:44.203577 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:13:44.203649 2492 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:44.203992 388 external_mini_cluster.cc:1428] Setting key a1448b088d9214e7fc238b6f730a0643
I20260619 13:13:44.204248 2492 heartbeater.cc:511] Master 127.0.97.62:39397 requested a full tablet report, sending...
I20260619 13:13:44.205365 2146 ts_manager.cc:194] Registered new tserver with Master: 2794dbae640c42cca0a0703181f52f09 (127.0.97.2:45659)
I20260619 13:13:44.206267 2146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45361
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:13:44.238019 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39397
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:44.533546 2496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:44.533798 2496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:44.533932 2496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:44.540688 2496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:44.540781 2496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:44.540866 2496 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 13:13:44.540915 2496 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:13:44.540980 2496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:13:44.550074 2496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:44.551209 2496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:44.552420 2496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:44.564206 2501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:44.564172 2504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:44.564149 2502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:44.564930 2496 server_base.cc:1034] running on GCE node
I20260619 13:13:44.565465 2496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:44.566398 2496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:44.567606 2496 hybrid_clock.cc:648] HybridClock initialized: now 1781874824567524 us; error 60 us; skew 500 ppm
I20260619 13:13:44.567900 2496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:44 dist-test-slave-mkt3 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:44.570736 2496 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:13:44.572050 2496 webserver.cc:460] Webserver started at http://127.0.97.3:33863/ using document root <none> and password file <none>
I20260619 13:13:44.572715 2496 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:44.572835 2496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:44.573118 2496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:44.576041 2496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "cde1f7c9a13f493286ffbea483aa6ba5"
format_stamp: "Formatted at 2026-06-19 13:13:44 on dist-test-slave-mkt3"
server_key: "2776ebe32344e037762c0605b249bc60"
server_key_iv: "1dd3415b08516d8b7f752ed99d748800"
server_key_version: "encryptionkey@0"
I20260619 13:13:44.576773 2496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cde1f7c9a13f493286ffbea483aa6ba5"
format_stamp: "Formatted at 2026-06-19 13:13:44 on dist-test-slave-mkt3"
server_key: "2776ebe32344e037762c0605b249bc60"
server_key_iv: "1dd3415b08516d8b7f752ed99d748800"
server_key_version: "encryptionkey@0"
I20260619 13:13:44.581892 2496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 13:13:44.585156 2511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:44.586478 2496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:13:44.586617 2496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "cde1f7c9a13f493286ffbea483aa6ba5"
format_stamp: "Formatted at 2026-06-19 13:13:44 on dist-test-slave-mkt3"
server_key: "2776ebe32344e037762c0605b249bc60"
server_key_iv: "1dd3415b08516d8b7f752ed99d748800"
server_key_version: "encryptionkey@0"
I20260619 13:13:44.586786 2496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:44.606132 2496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:44.609472 2496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:44.609711 2496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:44.612370 2496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:44.612501 2496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:44.612681 2496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:44.612814 2496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:13:44 dist-test-slave-mkt3 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:44.633419 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.612790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37681 (local address 127.0.97.62:39397)
0619 13:13:44.612959 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:44.612966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:44.614457 (+ 1491us) server_negotiation.cc:407] Connection header received
0619 13:13:44.616344 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:44.616350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:44.616432 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:44.616525 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:44.618493 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.619423 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.621865 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.622287 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.622425 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.626751 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:44.626758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:44.626763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:44.626805 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:44.629731 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:44.630642 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:44.630648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:44.630651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:44.630803 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:44.631324 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:44.631330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:44.631333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:44.631683 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:44.631828 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:44.632781 (+ 953us) server_negotiation.cc:299] Negotiation successful
0619 13:13:44.633026 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 13:13:44.634289 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.613480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39397 (local address 127.0.97.3:37681)
0619 13:13:44.614312 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:44.614347 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:44.616080 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:44.616753 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:44.616764 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:44.617440 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:44.618332 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.618352 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.619600 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.619604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.621713 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.621727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.622718 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.622723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.622966 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.623679 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:44.623720 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:44.626458 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:44.629890 (+ 3432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:44.629901 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:44.629918 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:44.630477 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:44.630909 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:44.630913 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:44.630920 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:44.631203 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:44.631819 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:44.631826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:44.632282 (+ 456us) client_negotiation.cc:769] Sending connection context
0619 13:13:44.632576 (+ 294us) client_negotiation.cc:240] Negotiation successful
0619 13:13:44.633034 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":375,"threads_started":1}
I20260619 13:13:44.651206 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.639527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48379 (local address 127.0.97.62:39397)
0619 13:13:44.639658 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:44.639662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:44.640190 (+ 528us) server_negotiation.cc:407] Connection header received
0619 13:13:44.640355 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:44.640359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:44.640412 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:44.640482 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:44.641318 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.642027 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.643276 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.643648 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.643754 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.645720 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:44.645725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:44.645728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:44.645756 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:44.648542 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:44.649131 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:44.649136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:44.649140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:44.649255 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:44.649646 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:44.649653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:44.649656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:44.650025 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:44.650170 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:44.650668 (+ 498us) server_negotiation.cc:299] Negotiation successful
0619 13:13:44.650869 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 13:13:44.651290 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.639583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39397 (local address 127.0.97.3:48379)
0619 13:13:44.640102 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:44.640117 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:44.640246 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:44.640514 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:44.640518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:44.640922 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:44.641190 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.641198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.642168 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.642173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.643146 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.643157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.643806 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.643812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.643958 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.644521 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:44.644537 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:44.645489 (+ 952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:44.648653 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:44.648658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:44.648666 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:44.649033 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:44.649355 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:44.649360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:44.649364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:44.649518 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:44.650166 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:44.650172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:44.650388 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 13:13:44.650562 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 13:13:44.650857 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":283,"threads_started":1}
I20260619 13:13:44.661250 2496 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46567
I20260619 13:13:44.661280 2630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46567 every 8 connection(s)
I20260619 13:13:44.662425 2496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:13:44.664153 2631 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39397
I20260619 13:13:44.664340 2631 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:44.664904 2631 heartbeater.cc:511] Master 127.0.97.62:39397 requested a full tablet report, sending...
I20260619 13:13:44.666097 2146 ts_manager.cc:194] Registered new tserver with Master: cde1f7c9a13f493286ffbea483aa6ba5 (127.0.97.3:46567)
I20260619 13:13:44.666903 2146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37681
I20260619 13:13:44.671859 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 2496
I20260619 13:13:44.671998 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:13:44.672430 388 external_mini_cluster.cc:1428] Setting key 0d5cc1c9096eca1d5c062c2f9863964a
I20260619 13:13:44.675204 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:13:44 dist-test-slave-mkt3 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260619 13:13:44.708601 2353 heartbeater.cc:503] Master 127.0.97.62:39397 was elected leader, sending a full tablet report...
Jun 19 13:13:44 dist-test-slave-mkt3 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:44.716029 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.703707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44510 (local address 127.0.97.62:39397)
0619 13:13:44.703918 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:44.703924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:44.704095 (+ 171us) server_negotiation.cc:407] Connection header received
0619 13:13:44.704243 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:44.704247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:44.704323 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:44.704400 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:44.705617 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.706361 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.707510 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.707927 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.708018 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.711565 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:44.711570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:44.711573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:44.711604 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:44.713936 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:44.714432 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:44.714435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:44.714438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:44.714556 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:44.714861 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:44.714865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:44.714867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:44.715110 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:44.715282 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:44.715521 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 13:13:44.715687 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:13:44.719503 2146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:13:44.725819 2146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:13:44.763130 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.736426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37237 (local address 127.0.0.1:39426)
0619 13:13:44.749387 (+ 12961us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:44.749442 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:44.749651 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:44.755261 (+ 5610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:44.755270 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:44.755307 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:44.755723 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.755736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.758436 (+ 2700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.758441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.760643 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.760654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.762060 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.762066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.762281 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.762344 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 13:13:44.762492 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 13:13:44.762672 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12657,"mutex_wait_us":1,"thread_start_us":21537,"threads_started":1}
I20260619 13:13:44.763971 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.736605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.0.97.1:37237)
0619 13:13:44.754323 (+ 17718us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:44.754333 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:44.754367 (+ 34us) server_negotiation.cc:407] Connection header received
0619 13:13:44.754473 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:44.754482 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:44.754771 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:44.754985 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:44.755945 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.758298 (+ 2353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.760783 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.761892 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.762028 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.762237 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:44.763537 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0619 13:13:44.763698 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17533,"thread_start_us":17401,"threads_started":1}
I20260619 13:13:44.765594 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.737021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45659 (local address 127.0.0.1:59674)
0619 13:13:44.758208 (+ 21187us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:44.758224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:44.758326 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:44.759275 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:44.759280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:44.759313 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:44.759747 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.759759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.761489 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.761494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.763423 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.763436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.764844 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.764850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.765011 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.765041 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 13:13:44.765127 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 13:13:44.765238 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21081,"mutex_wait_us":1,"thread_start_us":12833,"threads_started":1}
I20260619 13:13:44.765654 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.737196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.0.97.2:45659)
0619 13:13:44.757807 (+ 20611us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:44.757813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:44.758438 (+ 625us) server_negotiation.cc:407] Connection header received
0619 13:13:44.758595 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:44.758604 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:44.758876 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:44.759074 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:44.759927 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.761355 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.763609 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.764666 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.764783 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.764961 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:44.765173 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 13:13:44.765335 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20468,"thread_start_us":20367,"threads_started":1}
I20260619 13:13:44.767283 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.736590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46567 (local address 127.0.0.1:33024)
0619 13:13:44.757914 (+ 21324us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:44.757937 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:44.758058 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:44.759586 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:44.759589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:44.759611 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:44.759972 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.759984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.761757 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.761765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.764032 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.764043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.766565 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.766572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.766732 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.766762 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 13:13:44.766850 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 13:13:44.766955 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21182,"thread_start_us":21047,"threads_started":1}
I20260619 13:13:44.767577 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.737061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33024 (local address 127.0.97.3:46567)
0619 13:13:44.758749 (+ 21688us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:44.758757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:44.758792 (+ 35us) server_negotiation.cc:407] Connection header received
0619 13:13:44.758887 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:44.758894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:44.759133 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:44.759319 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:44.760202 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.761640 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.764961 (+ 3321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.766366 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.766480 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.766667 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:44.767098 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 13:13:44.767272 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21504,"thread_start_us":22833,"threads_started":1}
I20260619 13:13:44.768193 2285 tablet_service.cc:1467] Processing CreateTablet for tablet 9765cc0f00224d519e3c6a28b04b2265 (DEFAULT_TABLE table=test-table [id=cf6b46c0e03a494f85ab614621310819]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:44.769153 2422 tablet_service.cc:1467] Processing CreateTablet for tablet 9765cc0f00224d519e3c6a28b04b2265 (DEFAULT_TABLE table=test-table [id=cf6b46c0e03a494f85ab614621310819]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:44.769661 2285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9765cc0f00224d519e3c6a28b04b2265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:44.770319 2561 tablet_service.cc:1467] Processing CreateTablet for tablet 9765cc0f00224d519e3c6a28b04b2265 (DEFAULT_TABLE table=test-table [id=cf6b46c0e03a494f85ab614621310819]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:44.770774 2422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9765cc0f00224d519e3c6a28b04b2265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:44.771358 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9765cc0f00224d519e3c6a28b04b2265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:44.782505 2650 tablet_bootstrap.cc:492] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5: Bootstrap starting.
I20260619 13:13:44.784299 2651 tablet_bootstrap.cc:492] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052: Bootstrap starting.
I20260619 13:13:44.786043 2650 tablet_bootstrap.cc:654] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:44.787230 2650 log.cc:824] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:44.787514 2651 tablet_bootstrap.cc:654] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:44.788462 2651 log.cc:824] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:44.789628 2652 tablet_bootstrap.cc:492] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09: Bootstrap starting.
I20260619 13:13:44.790414 2650 tablet_bootstrap.cc:492] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5: No bootstrap required, opened a new log
I20260619 13:13:44.790666 2650 ts_tablet_manager.cc:1397] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260619 13:13:44.791106 2651 tablet_bootstrap.cc:492] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052: No bootstrap required, opened a new log
I20260619 13:13:44.791344 2651 ts_tablet_manager.cc:1397] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 13:13:44.792814 2652 tablet_bootstrap.cc:654] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:44.793918 2652 log.cc:824] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:44.796591 2652 tablet_bootstrap.cc:492] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09: No bootstrap required, opened a new log
I20260619 13:13:44.796824 2652 ts_tablet_manager.cc:1397] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260619 13:13:44.798740 2651 raft_consensus.cc:348] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } }
I20260619 13:13:44.799057 2651 raft_consensus.cc:374] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:44.799147 2651 raft_consensus.cc:729] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a8968dc803419d9eb8ca8e0cf2f052, State: Initialized, Role: FOLLOWER
I20260619 13:13:44.799686 2651 consensus_queue.cc:260] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } }
I20260619 13:13:44.800102 2650 raft_consensus.cc:348] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } }
I20260619 13:13:44.800487 2650 raft_consensus.cc:374] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:44.800580 2650 raft_consensus.cc:729] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde1f7c9a13f493286ffbea483aa6ba5, State: Initialized, Role: FOLLOWER
I20260619 13:13:44.801143 2650 consensus_queue.cc:260] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } }
I20260619 13:13:44.802524 2631 heartbeater.cc:503] Master 127.0.97.62:39397 was elected leader, sending a full tablet report...
I20260619 13:13:44.803560 2651 ts_tablet_manager.cc:1428] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260619 13:13:44.804291 2650 ts_tablet_manager.cc:1428] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260619 13:13:44.805896 2652 raft_consensus.cc:348] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } }
I20260619 13:13:44.806284 2652 raft_consensus.cc:374] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:44.806382 2652 raft_consensus.cc:729] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2794dbae640c42cca0a0703181f52f09, State: Initialized, Role: FOLLOWER
I20260619 13:13:44.806943 2652 consensus_queue.cc:260] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } }
I20260619 13:13:44.807901 2492 heartbeater.cc:503] Master 127.0.97.62:39397 was elected leader, sending a full tablet report...
I20260619 13:13:44.808441 2652 ts_tablet_manager.cc:1428] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260619 13:13:44.859431 2657 raft_consensus.cc:482] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:13:44.859716 2657 raft_consensus.cc:504] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } }
I20260619 13:13:44.861322 2657 leader_election.cc:288] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3a8968dc803419d9eb8ca8e0cf2f052 (127.0.97.1:37237), 2794dbae640c42cca0a0703181f52f09 (127.0.97.2:45659)
I20260619 13:13:44.867969 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.861789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52561 (local address 127.0.97.2:45659)
0619 13:13:44.862175 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:44.862180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:44.862200 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:13:44.862268 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:44.862273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:44.862350 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:44.862432 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:44.862845 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.863839 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.866096 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.867000 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.867116 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.867214 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:44.867622 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 13:13:44.867736 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20260619 13:13:44.868000 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.861746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45659 (local address 127.0.97.3:52561)
0619 13:13:44.861956 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:44.861980 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:44.862094 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:44.862484 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:44.862489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:44.862516 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:44.862722 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.862731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.863947 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.863951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.865946 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.865959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.867139 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.867145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.867465 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.867496 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:13:44.867592 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 13:13:44.867720 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":59}
I20260619 13:13:44.868201 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.862109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52973 (local address 127.0.97.1:37237)
0619 13:13:44.862286 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:44.862291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:44.862508 (+ 217us) server_negotiation.cc:407] Connection header received
0619 13:13:44.862659 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:44.862664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:44.862739 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:44.862818 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:13:44.863230 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.864124 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.866332 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.867377 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.867498 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.867591 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:44.867856 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 13:13:44.867978 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:13:44.868315 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.861768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37237 (local address 127.0.97.3:52973)
0619 13:13:44.862408 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:44.862425 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:44.862538 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:44.862841 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:44.862845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:44.862864 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:13:44.863107 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.863112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.864234 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.864242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.866193 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.866206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.867562 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.867568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.867720 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.867741 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 13:13:44.867818 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 13:13:44.867933 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":5,"spinlock_wait_cycles":7424,"thread_start_us":744,"threads_started":1}
I20260619 13:13:44.869246 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9765cc0f00224d519e3c6a28b04b2265" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" is_pre_election: true
I20260619 13:13:44.869256 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9765cc0f00224d519e3c6a28b04b2265" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794dbae640c42cca0a0703181f52f09" is_pre_election: true
I20260619 13:13:44.869756 2308 raft_consensus.cc:2454] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 0.
I20260619 13:13:44.869756 2442 raft_consensus.cc:2454] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 0.
I20260619 13:13:44.870437 2512 leader_election.cc:302] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2794dbae640c42cca0a0703181f52f09, c3a8968dc803419d9eb8ca8e0cf2f052, cde1f7c9a13f493286ffbea483aa6ba5; no voters:
I20260619 13:13:44.870801 2657 raft_consensus.cc:2781] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:13:44.871066 2657 raft_consensus.cc:482] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:13:44.871169 2657 raft_consensus.cc:3037] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:44.873378 2657 raft_consensus.cc:504] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } }
I20260619 13:13:44.873948 2657 leader_election.cc:288] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 election: Requested vote from peers c3a8968dc803419d9eb8ca8e0cf2f052 (127.0.97.1:37237), 2794dbae640c42cca0a0703181f52f09 (127.0.97.2:45659)
I20260619 13:13:44.874305 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9765cc0f00224d519e3c6a28b04b2265" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052"
I20260619 13:13:44.874343 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9765cc0f00224d519e3c6a28b04b2265" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794dbae640c42cca0a0703181f52f09"
I20260619 13:13:44.874454 2308 raft_consensus.cc:3037] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:44.874536 2442 raft_consensus.cc:3037] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:44.876636 2442 raft_consensus.cc:2454] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 1.
I20260619 13:13:44.876693 2308 raft_consensus.cc:2454] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 1.
I20260619 13:13:44.877049 2515 leader_election.cc:302] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2794dbae640c42cca0a0703181f52f09, c3a8968dc803419d9eb8ca8e0cf2f052, cde1f7c9a13f493286ffbea483aa6ba5; no voters:
I20260619 13:13:44.877354 2657 raft_consensus.cc:2781] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:44.877733 2657 raft_consensus.cc:686] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 LEADER]: Becoming Leader. State: Replica: cde1f7c9a13f493286ffbea483aa6ba5, State: Running, Role: LEADER
I20260619 13:13:44.878252 2657 consensus_queue.cc:237] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } }
I20260619 13:13:44.882143 2145 catalog_manager.cc:5495] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cde1f7c9a13f493286ffbea483aa6ba5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } health_report { overall_health: UNKNOWN } } }
W20260619 13:13:44.913926 2632 tablet.cc:2369] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 13:13:44 dist-test-slave-mkt3 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:44.939985 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.928010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39397 (local address 127.0.0.1:44518)
0619 13:13:44.928563 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:44.928578 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:44.928696 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:44.928955 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:44.928957 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:44.929610 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:44.929896 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.929906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.930916 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.930919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.931858 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:44.931868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.932564 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.932570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:44.932688 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.933309 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:44.933325 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:44.935370 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:44.938009 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:44.938019 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:44.938025 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:44.938398 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:44.938710 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:44.938714 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:44.938716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:44.938839 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:44.939228 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:44.939234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:44.939381 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 13:13:44.939510 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 13:13:44.939693 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":286,"threads_started":1}
I20260619 13:13:44.940040 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:44.928010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44518 (local address 127.0.97.62:39397)
0619 13:13:44.928186 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:44.928190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:44.928655 (+ 465us) server_negotiation.cc:407] Connection header received
0619 13:13:44.928785 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:44.928789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:44.928846 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:44.928925 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:44.930036 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.930802 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.931973 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:44.932381 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:44.932471 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:44.935528 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:44.935533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:44.935536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:44.935563 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:44.937902 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:44.938494 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:44.938498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:44.938500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:44.938630 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:44.938917 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:44.938920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:44.938922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:44.939139 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:44.939235 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:44.939588 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 13:13:44.939786 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":45}
I20260619 13:13:44.944595 2145 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44518:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 13:13:44.945567 2145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:13:44.952378 2561 tablet_service.cc:1467] Processing CreateTablet for tablet 4448d2d68b5e419e8d7d1df347835793 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37abb5d734b446b787046660003aa842]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:13:44.952229 2285 tablet_service.cc:1467] Processing CreateTablet for tablet 4448d2d68b5e419e8d7d1df347835793 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37abb5d734b446b787046660003aa842]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:13:44.953194 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4448d2d68b5e419e8d7d1df347835793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:44.953208 2285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4448d2d68b5e419e8d7d1df347835793. 1 dirs total, 0 dirs full, 0 dirs failed
W20260619 13:13:44.953208 2354 tablet.cc:2369] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:13:44.952366 2422 tablet_service.cc:1467] Processing CreateTablet for tablet 4448d2d68b5e419e8d7d1df347835793 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37abb5d734b446b787046660003aa842]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:13:44.954504 2422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4448d2d68b5e419e8d7d1df347835793. 1 dirs total, 0 dirs full, 0 dirs failed
W20260619 13:13:44.958097 2493 tablet.cc:2369] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:13:44.959286 2652 tablet_bootstrap.cc:492] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09: Bootstrap starting.
I20260619 13:13:44.959932 2650 tablet_bootstrap.cc:492] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5: Bootstrap starting.
I20260619 13:13:44.960817 2652 tablet_bootstrap.cc:654] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:44.961359 2651 tablet_bootstrap.cc:492] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052: Bootstrap starting.
I20260619 13:13:44.961397 2650 tablet_bootstrap.cc:654] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:44.962836 2651 tablet_bootstrap.cc:654] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:44.963320 2652 tablet_bootstrap.cc:492] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09: No bootstrap required, opened a new log
I20260619 13:13:44.963459 2652 ts_tablet_manager.cc:1397] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260619 13:13:44.963737 2650 tablet_bootstrap.cc:492] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5: No bootstrap required, opened a new log
I20260619 13:13:44.963883 2650 ts_tablet_manager.cc:1397] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260619 13:13:44.964293 2652 raft_consensus.cc:348] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:44.964529 2652 raft_consensus.cc:374] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:44.964628 2652 raft_consensus.cc:729] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2794dbae640c42cca0a0703181f52f09, State: Initialized, Role: FOLLOWER
I20260619 13:13:44.964596 2650 raft_consensus.cc:348] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:44.964809 2651 tablet_bootstrap.cc:492] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052: No bootstrap required, opened a new log
I20260619 13:13:44.964811 2650 raft_consensus.cc:374] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:44.964892 2650 raft_consensus.cc:729] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde1f7c9a13f493286ffbea483aa6ba5, State: Initialized, Role: FOLLOWER
I20260619 13:13:44.964936 2651 ts_tablet_manager.cc:1397] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 13:13:44.964843 2652 consensus_queue.cc:260] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:44.965123 2650 consensus_queue.cc:260] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:44.965253 2658 tablet_replica.cc:439] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } }
I20260619 13:13:44.965584 2658 tablet_replica.cc:442] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:13:44.965492 2657 tablet_replica.cc:439] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } }
I20260619 13:13:44.965571 2651 raft_consensus.cc:348] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:44.965734 2657 tablet_replica.cc:442] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:13:44.965791 2651 raft_consensus.cc:374] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:44.965896 2651 raft_consensus.cc:729] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a8968dc803419d9eb8ca8e0cf2f052, State: Initialized, Role: FOLLOWER
I20260619 13:13:44.966097 2651 consensus_queue.cc:260] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:44.966492 2656 tablet_replica.cc:439] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } }
I20260619 13:13:44.966725 2650 ts_tablet_manager.cc:1428] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 13:13:44.966817 2656 tablet_replica.cc:442] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:13:44.967245 2651 ts_tablet_manager.cc:1428] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 13:13:44.968883 2652 ts_tablet_manager.cc:1428] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260619 13:13:45.121882 2657 raft_consensus.cc:482] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:13:45.122092 2657 raft_consensus.cc:504] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:45.122742 2657 leader_election.cc:288] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3a8968dc803419d9eb8ca8e0cf2f052 (127.0.97.1:37237), 2794dbae640c42cca0a0703181f52f09 (127.0.97.2:45659)
I20260619 13:13:45.123268 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448d2d68b5e419e8d7d1df347835793" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" is_pre_election: true
I20260619 13:13:45.123272 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448d2d68b5e419e8d7d1df347835793" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794dbae640c42cca0a0703181f52f09" is_pre_election: true
I20260619 13:13:45.123499 2308 raft_consensus.cc:2454] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 0.
I20260619 13:13:45.123497 2442 raft_consensus.cc:2454] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 0.
I20260619 13:13:45.123883 2512 leader_election.cc:302] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3a8968dc803419d9eb8ca8e0cf2f052, cde1f7c9a13f493286ffbea483aa6ba5; no voters:
I20260619 13:13:45.124168 2657 raft_consensus.cc:2781] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:13:45.124249 2657 raft_consensus.cc:482] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:13:45.124336 2657 raft_consensus.cc:3037] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:45.126299 2657 raft_consensus.cc:504] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:45.126813 2657 leader_election.cc:288] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 election: Requested vote from peers c3a8968dc803419d9eb8ca8e0cf2f052 (127.0.97.1:37237), 2794dbae640c42cca0a0703181f52f09 (127.0.97.2:45659)
I20260619 13:13:45.127243 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448d2d68b5e419e8d7d1df347835793" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052"
I20260619 13:13:45.127243 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4448d2d68b5e419e8d7d1df347835793" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794dbae640c42cca0a0703181f52f09"
I20260619 13:13:45.127456 2442 raft_consensus.cc:3037] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:45.127456 2308 raft_consensus.cc:3037] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:45.130296 2442 raft_consensus.cc:2454] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 1.
I20260619 13:13:45.130296 2308 raft_consensus.cc:2454] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 1.
I20260619 13:13:45.130686 2515 leader_election.cc:302] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2794dbae640c42cca0a0703181f52f09, cde1f7c9a13f493286ffbea483aa6ba5; no voters:
I20260619 13:13:45.130910 2657 raft_consensus.cc:2781] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:45.131085 2657 raft_consensus.cc:686] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 LEADER]: Becoming Leader. State: Replica: cde1f7c9a13f493286ffbea483aa6ba5, State: Running, Role: LEADER
I20260619 13:13:45.131326 2657 consensus_queue.cc:237] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:45.132815 2660 tablet_replica.cc:439] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5: TxnStatusTablet state changed. Reason: New leader cde1f7c9a13f493286ffbea483aa6ba5. Latest consensus state: current_term: 1 leader_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } }
I20260619 13:13:45.133128 2660 tablet_replica.cc:442] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:13:45.133716 2672 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 13:13:45.133976 2672 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 13:13:45.134106 2145 catalog_manager.cc:5495] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cde1f7c9a13f493286ffbea483aa6ba5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } health_report { overall_health: HEALTHY } } }
I20260619 13:13:45.166738 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:45.161513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.0.97.3:46567)
0619 13:13:45.161720 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:45.161725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:45.161743 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:13:45.162052 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:45.162056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:45.162133 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:45.162256 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:13:45.162609 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:45.163309 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:45.164509 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:45.165043 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:45.165147 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:45.165588 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:13:45.166160 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:13:45.166289 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:45.166373 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 13:13:45.166448 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 13:13:45.166783 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:45.161438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46567 (local address 127.0.0.1:33026)
0619 13:13:45.161666 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:45.161687 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:45.161820 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:45.162287 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:45.162292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:45.162304 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:13:45.162490 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:45.162498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:45.163421 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:45.163426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:45.164381 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:45.164392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:45.165184 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:45.165191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:45.165354 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:45.165410 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:13:45.166259 (+ 849us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:13:45.166280 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 13:13:45.166355 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 13:13:45.166512 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 13:13:45.174500 2308 raft_consensus.cc:1261] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Refusing update from remote peer cde1f7c9a13f493286ffbea483aa6ba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:13:45.174500 2442 raft_consensus.cc:1261] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Refusing update from remote peer cde1f7c9a13f493286ffbea483aa6ba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:13:45.175437 2660 consensus_queue.cc:1035] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:13:45.175786 2657 consensus_queue.cc:1035] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:13:45.188025 2656 tablet_replica.cc:439] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052: TxnStatusTablet state changed. Reason: New leader cde1f7c9a13f493286ffbea483aa6ba5. Latest consensus state: current_term: 1 leader_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } }
I20260619 13:13:45.188340 2656 tablet_replica.cc:442] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:13:45.191213 2657 tablet_replica.cc:439] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } }
I20260619 13:13:45.191522 2657 tablet_replica.cc:442] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:13:45.194370 2656 tablet_replica.cc:439] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } }
I20260619 13:13:45.194594 2656 tablet_replica.cc:442] T 4448d2d68b5e419e8d7d1df347835793 P c3a8968dc803419d9eb8ca8e0cf2f052: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:13:45.196798 2658 tablet_replica.cc:439] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09: TxnStatusTablet state changed. Reason: New leader cde1f7c9a13f493286ffbea483aa6ba5. Latest consensus state: current_term: 1 leader_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } }
I20260619 13:13:45.197070 2658 tablet_replica.cc:442] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:13:45.198479 2660 tablet_replica.cc:439] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } }
I20260619 13:13:45.198678 2660 tablet_replica.cc:442] T 4448d2d68b5e419e8d7d1df347835793 P cde1f7c9a13f493286ffbea483aa6ba5: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:13:45.200412 2677 mvcc.cc:205] Tried to move back new op lower bound from 7298559283905556480 to 7298559283740049408. Current Snapshot: MvccSnapshot[applied={T|T < 7298559283905556480}]
I20260619 13:13:45.203365 2658 tablet_replica.cc:439] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } } }
I20260619 13:13:45.203576 2658 tablet_replica.cc:442] T 4448d2d68b5e419e8d7d1df347835793 P 2794dbae640c42cca0a0703181f52f09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:13:45.207788 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:45.201662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.0.97.3:46567)
0619 13:13:45.201817 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:45.201823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:45.201862 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:13:45.201916 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:45.201921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:45.201997 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:45.202078 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:13:45.202636 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:45.203736 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:45.205812 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:45.206401 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:45.206537 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:45.207085 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:13:45.207197 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:13:45.207275 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:45.207441 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 13:13:45.207514 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 13:13:45.226414 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:45.218630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41331 (local address 127.0.97.3:46567)
0619 13:13:45.218797 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:45.218803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:45.221808 (+ 3005us) server_negotiation.cc:407] Connection header received
0619 13:13:45.221989 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:45.221994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:45.222053 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:45.222124 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:13:45.222671 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:45.223382 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:45.224799 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:45.225199 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:45.225301 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:45.225704 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:13:45.225815 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:13:45.225910 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:45.226037 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 13:13:45.226104 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":50}
I20260619 13:13:45.226503 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:45.218633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46567 (local address 127.0.97.3:41331)
0619 13:13:45.221710 (+ 3077us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:45.221726 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:45.221878 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:45.222146 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:45.222150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:45.222160 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:13:45.222529 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:45.222538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:45.223511 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:45.223515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:45.224681 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:45.224692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:45.225365 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:45.225370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:45.225480 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:45.225530 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:13:45.225917 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:13:45.225927 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 13:13:45.225994 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 13:13:45.226129 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2955,"mutex_wait_us":19,"thread_start_us":2845,"threads_started":1}
I20260619 13:13:45.235961 2442 raft_consensus.cc:1261] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Refusing update from remote peer cde1f7c9a13f493286ffbea483aa6ba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:13:45.236188 2307 raft_consensus.cc:1261] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Refusing update from remote peer cde1f7c9a13f493286ffbea483aa6ba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:13:45.236470 2657 consensus_queue.cc:1035] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:13:45.237159 2657 consensus_queue.cc:1035] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:13:45.302917 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:13:45.303145 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:13:45.306105 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de: Sending DeleteTablet for 3 replicas of tablet 9765cc0f00224d519e3c6a28b04b2265
I20260619 13:13:45.307070 2422 tablet_service.cc:1514] Processing DeleteTablet for tablet 9765cc0f00224d519e3c6a28b04b2265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59674
I20260619 13:13:45.307077 2285 tablet_service.cc:1514] Processing DeleteTablet for tablet 9765cc0f00224d519e3c6a28b04b2265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39426
I20260619 13:13:45.307756 2561 tablet_service.cc:1514] Processing DeleteTablet for tablet 9765cc0f00224d519e3c6a28b04b2265 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33024
I20260619 13:13:45.307950 2692 tablet_replica.cc:330] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052: stopping tablet replica
I20260619 13:13:45.308409 2692 raft_consensus.cc:2229] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:13:45.308822 2692 raft_consensus.cc:2258] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:45.310806 2692 ts_tablet_manager.cc:1905] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:45.313344 2145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:13:45.313997 2145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:13:45.318774 2692 ts_tablet_manager.cc:1918] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:13:45.319024 2692 log.cc:1196] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/9765cc0f00224d519e3c6a28b04b2265
I20260619 13:13:45.319585 2692 ts_tablet_manager.cc:1939] T 9765cc0f00224d519e3c6a28b04b2265 P c3a8968dc803419d9eb8ca8e0cf2f052: Deleting consensus metadata
I20260619 13:13:45.320655 2691 tablet_replica.cc:330] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09: stopping tablet replica
I20260619 13:13:45.321081 2691 raft_consensus.cc:2229] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:13:45.321468 2691 raft_consensus.cc:2258] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:45.321570 2421 tablet_service.cc:1467] Processing CreateTablet for tablet c8e17e6b655e4859b150828bd1468a14 (DEFAULT_TABLE table=test-table [id=d310a89f890b4ce6bf3431bc4c429441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:45.322084 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e17e6b655e4859b150828bd1468a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:45.322338 2560 tablet_service.cc:1467] Processing CreateTablet for tablet c8e17e6b655e4859b150828bd1468a14 (DEFAULT_TABLE table=test-table [id=d310a89f890b4ce6bf3431bc4c429441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:45.322748 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e17e6b655e4859b150828bd1468a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:45.323472 2130 catalog_manager.cc:4841] TS c3a8968dc803419d9eb8ca8e0cf2f052 (127.0.97.1:37237): tablet 9765cc0f00224d519e3c6a28b04b2265 (table test-table [id=cf6b46c0e03a494f85ab614621310819]) successfully deleted
I20260619 13:13:45.327519 2691 ts_tablet_manager.cc:1905] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:45.328330 2285 tablet_service.cc:1467] Processing CreateTablet for tablet c8e17e6b655e4859b150828bd1468a14 (DEFAULT_TABLE table=test-table [id=d310a89f890b4ce6bf3431bc4c429441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:45.328732 2285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e17e6b655e4859b150828bd1468a14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:45.329370 2650 tablet_bootstrap.cc:492] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5: Bootstrap starting.
I20260619 13:13:45.331171 2650 tablet_bootstrap.cc:654] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:45.331998 2693 tablet_replica.cc:330] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5: stopping tablet replica
I20260619 13:13:45.332437 2693 raft_consensus.cc:2229] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:13:45.333096 2693 raft_consensus.cc:2258] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:45.335072 2693 ts_tablet_manager.cc:1905] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:45.335429 2652 tablet_bootstrap.cc:492] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09: Bootstrap starting.
I20260619 13:13:45.337083 2652 tablet_bootstrap.cc:654] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:45.341073 2691 ts_tablet_manager.cc:1918] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:13:45.341236 2691 log.cc:1196] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/9765cc0f00224d519e3c6a28b04b2265
I20260619 13:13:45.341728 2691 ts_tablet_manager.cc:1939] T 9765cc0f00224d519e3c6a28b04b2265 P 2794dbae640c42cca0a0703181f52f09: Deleting consensus metadata
I20260619 13:13:45.342375 2651 tablet_bootstrap.cc:492] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052: Bootstrap starting.
I20260619 13:13:45.343973 2651 tablet_bootstrap.cc:654] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:45.344952 2133 catalog_manager.cc:4841] TS 2794dbae640c42cca0a0703181f52f09 (127.0.97.2:45659): tablet 9765cc0f00224d519e3c6a28b04b2265 (table test-table [id=cf6b46c0e03a494f85ab614621310819]) successfully deleted
I20260619 13:13:45.350793 2693 ts_tablet_manager.cc:1918] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:13:45.350968 2693 log.cc:1196] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/9765cc0f00224d519e3c6a28b04b2265
I20260619 13:13:45.351547 2693 ts_tablet_manager.cc:1939] T 9765cc0f00224d519e3c6a28b04b2265 P cde1f7c9a13f493286ffbea483aa6ba5: Deleting consensus metadata
I20260619 13:13:45.353370 2131 catalog_manager.cc:4841] TS cde1f7c9a13f493286ffbea483aa6ba5 (127.0.97.3:46567): tablet 9765cc0f00224d519e3c6a28b04b2265 (table test-table [id=cf6b46c0e03a494f85ab614621310819]) successfully deleted
I20260619 13:13:45.355628 2651 tablet_bootstrap.cc:492] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052: No bootstrap required, opened a new log
I20260619 13:13:45.355800 2651 ts_tablet_manager.cc:1397] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.002s
I20260619 13:13:45.356535 2651 raft_consensus.cc:348] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:45.356774 2651 raft_consensus.cc:374] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:45.356850 2651 raft_consensus.cc:729] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a8968dc803419d9eb8ca8e0cf2f052, State: Initialized, Role: FOLLOWER
I20260619 13:13:45.357087 2651 consensus_queue.cc:260] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:45.358831 2651 ts_tablet_manager.cc:1428] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 13:13:45.360224 2652 tablet_bootstrap.cc:492] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09: No bootstrap required, opened a new log
I20260619 13:13:45.360383 2652 ts_tablet_manager.cc:1397] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.011s
I20260619 13:13:45.361091 2652 raft_consensus.cc:348] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:45.361320 2652 raft_consensus.cc:374] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:45.361392 2652 raft_consensus.cc:729] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2794dbae640c42cca0a0703181f52f09, State: Initialized, Role: FOLLOWER
I20260619 13:13:45.362403 2652 consensus_queue.cc:260] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:45.362753 2650 tablet_bootstrap.cc:492] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5: No bootstrap required, opened a new log
I20260619 13:13:45.362931 2650 ts_tablet_manager.cc:1397] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.018s
I20260619 13:13:45.363051 2652 ts_tablet_manager.cc:1428] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260619 13:13:45.363662 2650 raft_consensus.cc:348] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:45.363909 2650 raft_consensus.cc:374] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:45.364001 2650 raft_consensus.cc:729] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde1f7c9a13f493286ffbea483aa6ba5, State: Initialized, Role: FOLLOWER
I20260619 13:13:45.364269 2650 consensus_queue.cc:260] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:45.364832 2650 ts_tablet_manager.cc:1428] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 13:13:45.608193 2687 raft_consensus.cc:482] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:13:45.608387 2687 raft_consensus.cc:504] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:45.609067 2687 leader_election.cc:288] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3a8968dc803419d9eb8ca8e0cf2f052 (127.0.97.1:37237), 2794dbae640c42cca0a0703181f52f09 (127.0.97.2:45659)
I20260619 13:13:45.609633 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e17e6b655e4859b150828bd1468a14" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794dbae640c42cca0a0703181f52f09" is_pre_election: true
I20260619 13:13:45.609634 2306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e17e6b655e4859b150828bd1468a14" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" is_pre_election: true
I20260619 13:13:45.609891 2306 raft_consensus.cc:2454] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 0.
I20260619 13:13:45.609891 2442 raft_consensus.cc:2454] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 0.
I20260619 13:13:45.610312 2512 leader_election.cc:302] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3a8968dc803419d9eb8ca8e0cf2f052, cde1f7c9a13f493286ffbea483aa6ba5; no voters:
I20260619 13:13:45.610589 2687 raft_consensus.cc:2781] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:13:45.610723 2687 raft_consensus.cc:482] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:13:45.610818 2687 raft_consensus.cc:3037] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:45.613976 2687 raft_consensus.cc:504] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:45.614609 2687 leader_election.cc:288] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 election: Requested vote from peers c3a8968dc803419d9eb8ca8e0cf2f052 (127.0.97.1:37237), 2794dbae640c42cca0a0703181f52f09 (127.0.97.2:45659)
I20260619 13:13:45.615072 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e17e6b655e4859b150828bd1468a14" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794dbae640c42cca0a0703181f52f09"
I20260619 13:13:45.615082 2306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e17e6b655e4859b150828bd1468a14" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052"
I20260619 13:13:45.615271 2442 raft_consensus.cc:3037] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:45.615278 2306 raft_consensus.cc:3037] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:45.617280 2442 raft_consensus.cc:2454] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 1.
I20260619 13:13:45.617689 2515 leader_election.cc:302] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2794dbae640c42cca0a0703181f52f09, cde1f7c9a13f493286ffbea483aa6ba5; no voters:
I20260619 13:13:45.617985 2687 raft_consensus.cc:2781] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:45.618127 2687 raft_consensus.cc:686] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 LEADER]: Becoming Leader. State: Replica: cde1f7c9a13f493286ffbea483aa6ba5, State: Running, Role: LEADER
I20260619 13:13:45.618111 2306 raft_consensus.cc:2454] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 1.
I20260619 13:13:45.618383 2687 consensus_queue.cc:237] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:45.620922 2143 catalog_manager.cc:5495] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cde1f7c9a13f493286ffbea483aa6ba5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } health_report { overall_health: HEALTHY } } }
I20260619 13:13:45.651230 2307 raft_consensus.cc:1261] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Refusing update from remote peer cde1f7c9a13f493286ffbea483aa6ba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:13:45.651751 2442 raft_consensus.cc:1261] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Refusing update from remote peer cde1f7c9a13f493286ffbea483aa6ba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:13:45.651733 2657 consensus_queue.cc:1035] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:13:45.652257 2657 consensus_queue.cc:1035] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:13:45.655157 2675 mvcc.cc:205] Tried to move back new op lower bound from 7298559285861584896 to 7298559285735178240. Current Snapshot: MvccSnapshot[applied={T|T < 7298559285861584896}]
I20260619 13:13:45.656188 2676 mvcc.cc:205] Tried to move back new op lower bound from 7298559285861584896 to 7298559285735178240. Current Snapshot: MvccSnapshot[applied={T|T < 7298559285861584896}]
W20260619 13:13:45.679567 2675 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 13:13:47.264750 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:13:47.264950 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:13:47.269044 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de: Sending DeleteTablet for 3 replicas of tablet c8e17e6b655e4859b150828bd1468a14
I20260619 13:13:47.269802 2285 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e17e6b655e4859b150828bd1468a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39426
I20260619 13:13:47.269867 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e17e6b655e4859b150828bd1468a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59674
I20260619 13:13:47.269961 2561 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e17e6b655e4859b150828bd1468a14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33024
I20260619 13:13:47.280607 2719 tablet_replica.cc:330] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052: stopping tablet replica
I20260619 13:13:47.280879 2719 raft_consensus.cc:2229] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:13:47.281193 2719 raft_consensus.cc:2258] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:47.281805 2721 tablet_replica.cc:330] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5: stopping tablet replica
I20260619 13:13:47.281822 2720 tablet_replica.cc:330] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09: stopping tablet replica
I20260619 13:13:47.282006 2721 raft_consensus.cc:2229] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:13:47.282006 2720 raft_consensus.cc:2229] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:13:47.282208 2720 raft_consensus.cc:2258] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:47.282286 2719 ts_tablet_manager.cc:1905] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:47.282343 2721 raft_consensus.cc:2258] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:47.283047 2720 ts_tablet_manager.cc:1905] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:47.283176 2721 ts_tablet_manager.cc:1905] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:47.287405 2719 ts_tablet_manager.cc:1918] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:13:47.287590 2719 log.cc:1196] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/c8e17e6b655e4859b150828bd1468a14
I20260619 13:13:47.288012 2719 ts_tablet_manager.cc:1939] T c8e17e6b655e4859b150828bd1468a14 P c3a8968dc803419d9eb8ca8e0cf2f052: Deleting consensus metadata
I20260619 13:13:47.289093 2130 catalog_manager.cc:4841] TS c3a8968dc803419d9eb8ca8e0cf2f052 (127.0.97.1:37237): tablet c8e17e6b655e4859b150828bd1468a14 (table test-table [id=d310a89f890b4ce6bf3431bc4c429441]) successfully deleted
I20260619 13:13:47.289204 2720 ts_tablet_manager.cc:1918] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:13:47.289323 2720 log.cc:1196] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/c8e17e6b655e4859b150828bd1468a14
I20260619 13:13:47.289664 2721 ts_tablet_manager.cc:1918] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:13:47.289762 2720 ts_tablet_manager.cc:1939] T c8e17e6b655e4859b150828bd1468a14 P 2794dbae640c42cca0a0703181f52f09: Deleting consensus metadata
I20260619 13:13:47.289793 2721 log.cc:1196] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/c8e17e6b655e4859b150828bd1468a14
I20260619 13:13:47.290228 2721 ts_tablet_manager.cc:1939] T c8e17e6b655e4859b150828bd1468a14 P cde1f7c9a13f493286ffbea483aa6ba5: Deleting consensus metadata
I20260619 13:13:47.290968 2133 catalog_manager.cc:4841] TS 2794dbae640c42cca0a0703181f52f09 (127.0.97.2:45659): tablet c8e17e6b655e4859b150828bd1468a14 (table test-table [id=d310a89f890b4ce6bf3431bc4c429441]) successfully deleted
I20260619 13:13:47.291317 2131 catalog_manager.cc:4841] TS cde1f7c9a13f493286ffbea483aa6ba5 (127.0.97.3:46567): tablet c8e17e6b655e4859b150828bd1468a14 (table test-table [id=d310a89f890b4ce6bf3431bc4c429441]) successfully deleted
Jun 19 13:13:47 dist-test-slave-mkt3 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874827, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 13:13:47 dist-test-slave-mkt3 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874827, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:47.322604 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:47.310359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44530 (local address 127.0.97.62:39397)
0619 13:13:47.310895 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:47.310900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:47.310921 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:13:47.311073 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:47.311077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:47.311134 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:47.311240 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:47.312379 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:47.313145 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:47.314429 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:47.314811 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:47.314908 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:47.318329 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:47.318334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:47.318337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:47.318367 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:47.320533 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:47.321090 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:47.321094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:47.321096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:47.321211 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:47.321483 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:47.321486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:47.321487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:47.321715 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:47.321916 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:47.322124 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 13:13:47.322294 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":273,"threads_started":1}
I20260619 13:13:47.325548 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:13:47.326122 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:13:47.331866 2285 tablet_service.cc:1467] Processing CreateTablet for tablet 255521127a2d41cc8d48c56d6b1bd37d (DEFAULT_TABLE table=test-table [id=ed34b0415a3b42e69527f8fcd5a17813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:47.331885 2421 tablet_service.cc:1467] Processing CreateTablet for tablet 255521127a2d41cc8d48c56d6b1bd37d (DEFAULT_TABLE table=test-table [id=ed34b0415a3b42e69527f8fcd5a17813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:47.332288 2285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 255521127a2d41cc8d48c56d6b1bd37d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:47.332319 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 255521127a2d41cc8d48c56d6b1bd37d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:47.332264 2561 tablet_service.cc:1467] Processing CreateTablet for tablet 255521127a2d41cc8d48c56d6b1bd37d (DEFAULT_TABLE table=test-table [id=ed34b0415a3b42e69527f8fcd5a17813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:47.332626 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 255521127a2d41cc8d48c56d6b1bd37d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:47.339237 2731 tablet_bootstrap.cc:492] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5: Bootstrap starting.
I20260619 13:13:47.339269 2732 tablet_bootstrap.cc:492] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052: Bootstrap starting.
I20260619 13:13:47.340502 2732 tablet_bootstrap.cc:654] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:47.341517 2731 tablet_bootstrap.cc:654] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:47.343727 2731 tablet_bootstrap.cc:492] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5: No bootstrap required, opened a new log
I20260619 13:13:47.343870 2731 ts_tablet_manager.cc:1397] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260619 13:13:47.344631 2731 raft_consensus.cc:348] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.344962 2731 raft_consensus.cc:374] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:47.345089 2731 raft_consensus.cc:729] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde1f7c9a13f493286ffbea483aa6ba5, State: Initialized, Role: FOLLOWER
I20260619 13:13:47.345358 2731 consensus_queue.cc:260] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.346143 2731 ts_tablet_manager.cc:1428] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 13:13:47.346997 2730 tablet_bootstrap.cc:492] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09: Bootstrap starting.
I20260619 13:13:47.348747 2730 tablet_bootstrap.cc:654] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:47.348857 2732 tablet_bootstrap.cc:492] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052: No bootstrap required, opened a new log
I20260619 13:13:47.348990 2732 ts_tablet_manager.cc:1397] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260619 13:13:47.349968 2732 raft_consensus.cc:348] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.350239 2732 raft_consensus.cc:374] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:47.350350 2732 raft_consensus.cc:729] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a8968dc803419d9eb8ca8e0cf2f052, State: Initialized, Role: FOLLOWER
I20260619 13:13:47.350611 2732 consensus_queue.cc:260] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.350737 2730 tablet_bootstrap.cc:492] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09: No bootstrap required, opened a new log
I20260619 13:13:47.350841 2730 ts_tablet_manager.cc:1397] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 13:13:47.351275 2732 ts_tablet_manager.cc:1428] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 13:13:47.351509 2730 raft_consensus.cc:348] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.351698 2730 raft_consensus.cc:374] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:47.351792 2730 raft_consensus.cc:729] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2794dbae640c42cca0a0703181f52f09, State: Initialized, Role: FOLLOWER
I20260619 13:13:47.352010 2730 consensus_queue.cc:260] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.352751 2730 ts_tablet_manager.cc:1428] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:13:47.365160 2687 raft_consensus.cc:482] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:13:47.365339 2687 raft_consensus.cc:504] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.365979 2687 leader_election.cc:288] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3a8968dc803419d9eb8ca8e0cf2f052 (127.0.97.1:37237), 2794dbae640c42cca0a0703181f52f09 (127.0.97.2:45659)
I20260619 13:13:47.366375 2306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255521127a2d41cc8d48c56d6b1bd37d" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" is_pre_election: true
I20260619 13:13:47.366462 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255521127a2d41cc8d48c56d6b1bd37d" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794dbae640c42cca0a0703181f52f09" is_pre_election: true
I20260619 13:13:47.366580 2306 raft_consensus.cc:2454] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 0.
I20260619 13:13:47.366703 2442 raft_consensus.cc:2454] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 0.
I20260619 13:13:47.366994 2512 leader_election.cc:302] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3a8968dc803419d9eb8ca8e0cf2f052, cde1f7c9a13f493286ffbea483aa6ba5; no voters:
I20260619 13:13:47.367269 2687 raft_consensus.cc:2781] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:13:47.367352 2687 raft_consensus.cc:482] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:13:47.367424 2687 raft_consensus.cc:3037] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:47.369256 2687 raft_consensus.cc:504] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.369736 2687 leader_election.cc:288] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 election: Requested vote from peers c3a8968dc803419d9eb8ca8e0cf2f052 (127.0.97.1:37237), 2794dbae640c42cca0a0703181f52f09 (127.0.97.2:45659)
I20260619 13:13:47.370103 2306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255521127a2d41cc8d48c56d6b1bd37d" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052"
I20260619 13:13:47.370187 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "255521127a2d41cc8d48c56d6b1bd37d" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794dbae640c42cca0a0703181f52f09"
I20260619 13:13:47.370285 2306 raft_consensus.cc:3037] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:47.370366 2442 raft_consensus.cc:3037] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:47.372628 2306 raft_consensus.cc:2454] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 1.
I20260619 13:13:47.372963 2512 leader_election.cc:302] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3a8968dc803419d9eb8ca8e0cf2f052, cde1f7c9a13f493286ffbea483aa6ba5; no voters:
I20260619 13:13:47.373032 2442 raft_consensus.cc:2454] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 1.
I20260619 13:13:47.373237 2687 raft_consensus.cc:2781] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:47.373397 2687 raft_consensus.cc:686] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 LEADER]: Becoming Leader. State: Replica: cde1f7c9a13f493286ffbea483aa6ba5, State: Running, Role: LEADER
I20260619 13:13:47.373613 2687 consensus_queue.cc:237] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.375947 2145 catalog_manager.cc:5495] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cde1f7c9a13f493286ffbea483aa6ba5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } health_report { overall_health: HEALTHY } } }
I20260619 13:13:47.398000 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:47.393113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.0.97.3:46567)
0619 13:13:47.393626 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:47.393630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:47.393647 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:13:47.393692 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:47.393695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:47.393754 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:47.393890 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:13:47.394172 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:47.395033 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:47.396363 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:47.396788 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:47.396883 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:47.397329 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:13:47.397443 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:13:47.397509 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:47.397604 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 13:13:47.397660 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":311,"threads_started":1}
I20260619 13:13:47.405799 2442 raft_consensus.cc:1261] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Refusing update from remote peer cde1f7c9a13f493286ffbea483aa6ba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:13:47.406230 2306 raft_consensus.cc:1261] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Refusing update from remote peer cde1f7c9a13f493286ffbea483aa6ba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:13:47.406301 2742 consensus_queue.cc:1035] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:13:47.406668 2702 consensus_queue.cc:1035] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:13:47.445322 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:13:47.445547 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:13:47.448029 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de: Sending DeleteTablet for 3 replicas of tablet 255521127a2d41cc8d48c56d6b1bd37d
I20260619 13:13:47.448698 2285 tablet_service.cc:1514] Processing DeleteTablet for tablet 255521127a2d41cc8d48c56d6b1bd37d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39426
I20260619 13:13:47.448786 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet 255521127a2d41cc8d48c56d6b1bd37d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33024
I20260619 13:13:47.448822 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet 255521127a2d41cc8d48c56d6b1bd37d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59674
I20260619 13:13:47.449014 2721 tablet_replica.cc:330] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5: stopping tablet replica
I20260619 13:13:47.449095 2720 tablet_replica.cc:330] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09: stopping tablet replica
I20260619 13:13:47.449121 2719 tablet_replica.cc:330] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052: stopping tablet replica
I20260619 13:13:47.449187 2721 raft_consensus.cc:2229] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:13:47.449257 2720 raft_consensus.cc:2229] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:13:47.449312 2719 raft_consensus.cc:2229] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:13:47.449411 2720 raft_consensus.cc:2258] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:47.449471 2721 raft_consensus.cc:2258] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:47.449492 2719 raft_consensus.cc:2258] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:47.449617 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:13:47.450310 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:13:47.450415 2721 ts_tablet_manager.cc:1905] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:47.450415 2720 ts_tablet_manager.cc:1905] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:47.450657 2719 ts_tablet_manager.cc:1905] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:47.456591 2720 ts_tablet_manager.cc:1918] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:13:47.456741 2720 log.cc:1196] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/255521127a2d41cc8d48c56d6b1bd37d
I20260619 13:13:47.456946 2285 tablet_service.cc:1467] Processing CreateTablet for tablet 30fc6880689c4180b00b55cb71f8ab31 (DEFAULT_TABLE table=test-table [id=12d7ac0b20ff4389b16b58fbeec93b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:47.457168 2720 ts_tablet_manager.cc:1939] T 255521127a2d41cc8d48c56d6b1bd37d P 2794dbae640c42cca0a0703181f52f09: Deleting consensus metadata
I20260619 13:13:47.457111 2421 tablet_service.cc:1467] Processing CreateTablet for tablet 30fc6880689c4180b00b55cb71f8ab31 (DEFAULT_TABLE table=test-table [id=12d7ac0b20ff4389b16b58fbeec93b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:47.457346 2285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fc6880689c4180b00b55cb71f8ab31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:47.457525 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fc6880689c4180b00b55cb71f8ab31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:47.458184 2133 catalog_manager.cc:4841] TS 2794dbae640c42cca0a0703181f52f09 (127.0.97.2:45659): tablet 255521127a2d41cc8d48c56d6b1bd37d (table test-table [id=ed34b0415a3b42e69527f8fcd5a17813]) successfully deleted
I20260619 13:13:47.461465 2732 tablet_bootstrap.cc:492] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052: Bootstrap starting.
I20260619 13:13:47.461884 2730 tablet_bootstrap.cc:492] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09: Bootstrap starting.
I20260619 13:13:47.461875 2719 ts_tablet_manager.cc:1918] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:13:47.462007 2719 log.cc:1196] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/255521127a2d41cc8d48c56d6b1bd37d
I20260619 13:13:47.462424 2719 ts_tablet_manager.cc:1939] T 255521127a2d41cc8d48c56d6b1bd37d P c3a8968dc803419d9eb8ca8e0cf2f052: Deleting consensus metadata
I20260619 13:13:47.462448 2560 tablet_service.cc:1467] Processing CreateTablet for tablet 30fc6880689c4180b00b55cb71f8ab31 (DEFAULT_TABLE table=test-table [id=12d7ac0b20ff4389b16b58fbeec93b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:13:47.462832 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fc6880689c4180b00b55cb71f8ab31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:47.463061 2732 tablet_bootstrap.cc:654] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:47.463387 2130 catalog_manager.cc:4841] TS c3a8968dc803419d9eb8ca8e0cf2f052 (127.0.97.1:37237): tablet 255521127a2d41cc8d48c56d6b1bd37d (table test-table [id=ed34b0415a3b42e69527f8fcd5a17813]) successfully deleted
I20260619 13:13:47.463871 2730 tablet_bootstrap.cc:654] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:47.467129 2721 ts_tablet_manager.cc:1918] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:13:47.467360 2721 log.cc:1196] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/255521127a2d41cc8d48c56d6b1bd37d
I20260619 13:13:47.467828 2721 ts_tablet_manager.cc:1939] T 255521127a2d41cc8d48c56d6b1bd37d P cde1f7c9a13f493286ffbea483aa6ba5: Deleting consensus metadata
I20260619 13:13:47.468986 2131 catalog_manager.cc:4841] TS cde1f7c9a13f493286ffbea483aa6ba5 (127.0.97.3:46567): tablet 255521127a2d41cc8d48c56d6b1bd37d (table test-table [id=ed34b0415a3b42e69527f8fcd5a17813]) successfully deleted
I20260619 13:13:47.469923 2732 tablet_bootstrap.cc:492] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052: No bootstrap required, opened a new log
I20260619 13:13:47.470081 2732 ts_tablet_manager.cc:1397] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 13:13:47.470502 2731 tablet_bootstrap.cc:492] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5: Bootstrap starting.
I20260619 13:13:47.470819 2730 tablet_bootstrap.cc:492] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09: No bootstrap required, opened a new log
I20260619 13:13:47.470778 2732 raft_consensus.cc:348] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.470957 2732 raft_consensus.cc:374] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:47.470957 2730 ts_tablet_manager.cc:1397] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260619 13:13:47.471002 2732 raft_consensus.cc:729] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a8968dc803419d9eb8ca8e0cf2f052, State: Initialized, Role: FOLLOWER
I20260619 13:13:47.471196 2732 consensus_queue.cc:260] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.471652 2732 ts_tablet_manager.cc:1428] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 13:13:47.471613 2730 raft_consensus.cc:348] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.471846 2730 raft_consensus.cc:374] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:47.471928 2730 raft_consensus.cc:729] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2794dbae640c42cca0a0703181f52f09, State: Initialized, Role: FOLLOWER
I20260619 13:13:47.471984 2731 tablet_bootstrap.cc:654] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:47.472160 2730 consensus_queue.cc:260] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.472754 2730 ts_tablet_manager.cc:1428] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 13:13:47.474867 2731 tablet_bootstrap.cc:492] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5: No bootstrap required, opened a new log
I20260619 13:13:47.474989 2731 ts_tablet_manager.cc:1397] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260619 13:13:47.476053 2731 raft_consensus.cc:348] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.476277 2731 raft_consensus.cc:374] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:47.476353 2731 raft_consensus.cc:729] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde1f7c9a13f493286ffbea483aa6ba5, State: Initialized, Role: FOLLOWER
I20260619 13:13:47.476583 2731 consensus_queue.cc:260] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.477066 2731 ts_tablet_manager.cc:1428] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 13:13:47.569008 2742 raft_consensus.cc:482] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:13:47.569213 2742 raft_consensus.cc:504] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.569785 2742 leader_election.cc:288] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3a8968dc803419d9eb8ca8e0cf2f052 (127.0.97.1:37237), 2794dbae640c42cca0a0703181f52f09 (127.0.97.2:45659)
I20260619 13:13:47.570370 2306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fc6880689c4180b00b55cb71f8ab31" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" is_pre_election: true
I20260619 13:13:47.570389 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fc6880689c4180b00b55cb71f8ab31" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794dbae640c42cca0a0703181f52f09" is_pre_election: true
I20260619 13:13:47.570643 2306 raft_consensus.cc:2454] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 0.
I20260619 13:13:47.570641 2441 raft_consensus.cc:2454] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 0.
I20260619 13:13:47.571077 2512 leader_election.cc:302] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2794dbae640c42cca0a0703181f52f09, c3a8968dc803419d9eb8ca8e0cf2f052, cde1f7c9a13f493286ffbea483aa6ba5; no voters:
I20260619 13:13:47.571372 2742 raft_consensus.cc:2781] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:13:47.571456 2742 raft_consensus.cc:482] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:13:47.571549 2742 raft_consensus.cc:3037] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:47.573490 2742 raft_consensus.cc:504] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.574048 2742 leader_election.cc:288] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 election: Requested vote from peers c3a8968dc803419d9eb8ca8e0cf2f052 (127.0.97.1:37237), 2794dbae640c42cca0a0703181f52f09 (127.0.97.2:45659)
I20260619 13:13:47.574401 2306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fc6880689c4180b00b55cb71f8ab31" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052"
I20260619 13:13:47.574460 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fc6880689c4180b00b55cb71f8ab31" candidate_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2794dbae640c42cca0a0703181f52f09"
I20260619 13:13:47.574575 2306 raft_consensus.cc:3037] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:47.574625 2441 raft_consensus.cc:3037] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:47.576468 2306 raft_consensus.cc:2454] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 1.
I20260619 13:13:47.576526 2441 raft_consensus.cc:2454] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde1f7c9a13f493286ffbea483aa6ba5 in term 1.
I20260619 13:13:47.576826 2512 leader_election.cc:302] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3a8968dc803419d9eb8ca8e0cf2f052, cde1f7c9a13f493286ffbea483aa6ba5; no voters:
I20260619 13:13:47.577046 2742 raft_consensus.cc:2781] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:47.577205 2742 raft_consensus.cc:686] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 LEADER]: Becoming Leader. State: Replica: cde1f7c9a13f493286ffbea483aa6ba5, State: Running, Role: LEADER
I20260619 13:13:47.577447 2742 consensus_queue.cc:237] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } }
I20260619 13:13:47.579859 2143 catalog_manager.cc:5495] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cde1f7c9a13f493286ffbea483aa6ba5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cde1f7c9a13f493286ffbea483aa6ba5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46567 } health_report { overall_health: HEALTHY } } }
I20260619 13:13:47.607721 2441 raft_consensus.cc:1261] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Refusing update from remote peer cde1f7c9a13f493286ffbea483aa6ba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:13:47.607749 2306 raft_consensus.cc:1261] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Refusing update from remote peer cde1f7c9a13f493286ffbea483aa6ba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:13:47.608191 2702 consensus_queue.cc:1035] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3a8968dc803419d9eb8ca8e0cf2f052" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:13:47.608443 2687 consensus_queue.cc:1035] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2794dbae640c42cca0a0703181f52f09" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 13:13:47.624639 2700 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 13:13:49.213980 2145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:13:49.214255 2145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:13:49.218463 2145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e42fb3e416e54d859259ae7196d677de: Sending DeleteTablet for 3 replicas of tablet 30fc6880689c4180b00b55cb71f8ab31
I20260619 13:13:49.219249 2285 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fc6880689c4180b00b55cb71f8ab31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39426
I20260619 13:13:49.219276 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fc6880689c4180b00b55cb71f8ab31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59674
I20260619 13:13:49.219393 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fc6880689c4180b00b55cb71f8ab31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:13:49 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33024
I20260619 13:13:49.220938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 2217
I20260619 13:13:49.221139 2773 tablet_replica.cc:330] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5: stopping tablet replica
I20260619 13:13:49.221323 2773 raft_consensus.cc:2229] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:13:49.221648 2773 raft_consensus.cc:2258] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:49.222651 2773 ts_tablet_manager.cc:1905] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:49.223165 2771 tablet_replica.cc:330] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052: stopping tablet replica
I20260619 13:13:49.223858 2772 tablet_replica.cc:330] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09: stopping tablet replica
I20260619 13:13:49.224025 2772 raft_consensus.cc:2229] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:13:49.224223 2772 raft_consensus.cc:2258] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:49.224522 2771 raft_consensus.cc:2229] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:13:49.225164 2772 ts_tablet_manager.cc:1905] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:49.226051 2771 raft_consensus.cc:2258] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:13:49.227025 2771 ts_tablet_manager.cc:1905] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:13:49.230672 2773 ts_tablet_manager.cc:1918] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:13:49.230872 2773 log.cc:1196] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/30fc6880689c4180b00b55cb71f8ab31
I20260619 13:13:49.230911 2772 ts_tablet_manager.cc:1918] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:13:49.231088 2772 log.cc:1196] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/30fc6880689c4180b00b55cb71f8ab31
I20260619 13:13:49.231375 2773 ts_tablet_manager.cc:1939] T 30fc6880689c4180b00b55cb71f8ab31 P cde1f7c9a13f493286ffbea483aa6ba5: Deleting consensus metadata
I20260619 13:13:49.231534 2772 ts_tablet_manager.cc:1939] T 30fc6880689c4180b00b55cb71f8ab31 P 2794dbae640c42cca0a0703181f52f09: Deleting consensus metadata
I20260619 13:13:49.232596 2131 catalog_manager.cc:4841] TS cde1f7c9a13f493286ffbea483aa6ba5 (127.0.97.3:46567): tablet 30fc6880689c4180b00b55cb71f8ab31 (table test-table [id=12d7ac0b20ff4389b16b58fbeec93b13]) successfully deleted
I20260619 13:13:49.232854 2133 catalog_manager.cc:4841] TS 2794dbae640c42cca0a0703181f52f09 (127.0.97.2:45659): tablet 30fc6880689c4180b00b55cb71f8ab31 (table test-table [id=12d7ac0b20ff4389b16b58fbeec93b13]) successfully deleted
I20260619 13:13:49.233333 2771 ts_tablet_manager.cc:1918] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 13:13:49.233438 2771 log.cc:1196] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/30fc6880689c4180b00b55cb71f8ab31
I20260619 13:13:49.233794 2771 ts_tablet_manager.cc:1939] T 30fc6880689c4180b00b55cb71f8ab31 P c3a8968dc803419d9eb8ca8e0cf2f052: Deleting consensus metadata
I20260619 13:13:49.234659 2130 catalog_manager.cc:4841] TS c3a8968dc803419d9eb8ca8e0cf2f052 (127.0.97.1:37237): tablet 30fc6880689c4180b00b55cb71f8ab31 (table test-table [id=12d7ac0b20ff4389b16b58fbeec93b13]) successfully deleted
Jun 19 13:13:49 dist-test-slave-mkt3 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874827, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:13:49.249523 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:49.221858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39440 (local address 127.0.97.1:37237)
0619 13:13:49.232906 (+ 11048us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:49.232912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:49.232937 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:13:49.232989 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:49.232993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:49.233071 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:49.233195 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:49.234539 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:49.235296 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:49.236456 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:49.236861 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:49.236969 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:49.244419 (+ 7450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:49.244432 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:49.244448 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:49.244478 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:49.247338 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:49.247929 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:49.247936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:49.247938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:49.248050 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:49.248343 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:49.248346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:49.248348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:49.248634 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:49.248838 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:49.249072 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 13:13:49.249228 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10902,"thread_start_us":10772,"threads_started":1}
W20260619 13:13:49.250114 2348 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39440
W20260619 13:13:49.250700 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 2217: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:13:49.250900 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 2217
I20260619 13:13:49.264850 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 2357
Jun 19 13:13:49 dist-test-slave-mkt3 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874827, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:13:49.278162 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:49.265521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.0.97.2:45659)
0619 13:13:49.265999 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:49.266003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:49.266018 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:13:49.266067 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:49.266070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:49.266123 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:49.266206 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:49.267451 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:49.268131 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:49.269298 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:49.269675 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:49.269762 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:49.273411 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:49.273424 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:49.273439 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:49.273471 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:49.276035 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:49.276619 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:49.276624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:49.276627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:49.276748 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:49.277004 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:49.277007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:49.277009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:49.277312 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:49.277502 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:49.277719 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 13:13:49.277893 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":239,"threads_started":1}
W20260619 13:13:49.278795 2487 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59682
W20260619 13:13:49.279346 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 2357: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:13:49.279475 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 2357
I20260619 13:13:49.292827 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 2496
Jun 19 13:13:49 dist-test-slave-mkt3 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874827, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:13:49.306924 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:49.293495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.0.97.3:46567)
0619 13:13:49.293958 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:49.293962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:49.293977 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:13:49.294026 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:49.294029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:49.294084 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:49.294171 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:49.295430 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:49.296212 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:49.297381 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:49.297778 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:49.297979 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:49.301790 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:49.301805 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:49.301821 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:49.301882 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:49.304446 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:49.305097 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:49.305103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:49.305106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:49.305216 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:49.305546 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:49.305549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:49.305551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:49.305981 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:49.306187 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:49.306415 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 13:13:49.306569 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":230,"threads_started":1}
W20260619 13:13:49.307544 2626 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33044
W20260619 13:13:49.308100 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 2496: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:13:49.308252 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 2496
I20260619 13:13:49.327500 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 2114
I20260619 13:13:49.328570 2186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:13:49.439190 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 2114
2026-06-19T13:13:49Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7869 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:13:49 dist-test-slave-mkt3 krb5kdc[2781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:13:49 dist-test-slave-mkt3 krb5kdc[2781](info): set up 2 sockets
Jun 19 13:13:49 dist-test-slave-mkt3 krb5kdc[2781](info): commencing operation
krb5kdc: starting...
W20260619 13:13:50.425803 388 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.002s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:13:50 dist-test-slave-mkt3 krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:13:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:13:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:13:50.587431 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46659
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:50.888011 2794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:50.888306 2794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:50.888432 2794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:50.895483 2794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:13:50.895589 2794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:50.895638 2794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:50.895714 2794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:13:50.895768 2794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:13:50.905340 2794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:50.906643 2794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:50.907932 2794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:50.919016 2799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:50.919070 2802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:50.919109 2800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:50.919633 2794 server_base.cc:1034] running on GCE node
I20260619 13:13:50.920380 2794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:50.921804 2794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:50.923040 2794 hybrid_clock.cc:648] HybridClock initialized: now 1781874830922985 us; error 43 us; skew 500 ppm
I20260619 13:13:50.923399 2794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:50 dist-test-slave-mkt3 krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:50.926406 2794 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:13:50.927796 2794 webserver.cc:460] Webserver started at http://127.0.97.62:39411/ using document root <none> and password file <none>
I20260619 13:13:50.928449 2794 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:50.928548 2794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:50.928839 2794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:50.931743 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "2f8dca51d3814c38b692eac3e66eff0f"
format_stamp: "Formatted at 2026-06-19 13:13:50 on dist-test-slave-mkt3"
server_key: "cd1fbc96a081568c99170dceb053827f"
server_key_iv: "728f58144b42fe6ae6d7f89bf3ac93ae"
server_key_version: "encryptionkey@0"
I20260619 13:13:50.932478 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "2f8dca51d3814c38b692eac3e66eff0f"
format_stamp: "Formatted at 2026-06-19 13:13:50 on dist-test-slave-mkt3"
server_key: "cd1fbc96a081568c99170dceb053827f"
server_key_iv: "728f58144b42fe6ae6d7f89bf3ac93ae"
server_key_version: "encryptionkey@0"
I20260619 13:13:50.937328 2794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:13:50.940506 2809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:50.941744 2794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 13:13:50.941932 2794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "2f8dca51d3814c38b692eac3e66eff0f"
format_stamp: "Formatted at 2026-06-19 13:13:50 on dist-test-slave-mkt3"
server_key: "cd1fbc96a081568c99170dceb053827f"
server_key_iv: "728f58144b42fe6ae6d7f89bf3ac93ae"
server_key_version: "encryptionkey@0"
I20260619 13:13:50.942092 2794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:50.971678 2794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:50.975731 2794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:50.975979 2794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:50.992677 2794 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46659
I20260619 13:13:50.992698 2860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46659 every 8 connection(s)
I20260619 13:13:50.994024 2794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:13:50.997531 2861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:51.002367 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 2794
I20260619 13:13:51.002573 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:13:51.003172 388 external_mini_cluster.cc:1428] Setting key e73596bc8aab7ca6b33d27e49a79a855
I20260619 13:13:51.007992 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f: Bootstrap starting.
I20260619 13:13:51.011423 2861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:51.012606 2861 log.cc:824] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:51.015748 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f: No bootstrap required, opened a new log
Jun 19 13:13:51 dist-test-slave-mkt3 krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:51.024399 2861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8dca51d3814c38b692eac3e66eff0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46659 } }
I20260619 13:13:51.024885 2861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:51.024983 2861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8dca51d3814c38b692eac3e66eff0f, State: Initialized, Role: FOLLOWER
I20260619 13:13:51.025580 2861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8dca51d3814c38b692eac3e66eff0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46659 } }
I20260619 13:13:51.025801 2861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:13:51.025895 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:51.005065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.0.97.62:46659)
0619 13:13:51.005913 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:51.005925 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:51.005971 (+ 46us) server_negotiation.cc:407] Connection header received
0619 13:13:51.007617 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:51.007651 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:51.008053 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:51.008405 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:51.010124 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:51.011393 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:51.013082 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:51.014099 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:51.014245 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:51.018692 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:51.018700 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:51.018720 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:51.018756 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:51.021998 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:51.022787 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:51.022794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:51.022800 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:51.022977 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:51.023335 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:51.023338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:51.023340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:51.023854 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:51.024078 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:51.024449 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 13:13:51.024776 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":360,"threads_started":1}
I20260619 13:13:51.025956 2861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:13:51.026147 2861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:51.028654 2861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8dca51d3814c38b692eac3e66eff0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46659 } }
I20260619 13:13:51.029158 2861 leader_election.cc:302] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f8dca51d3814c38b692eac3e66eff0f; no voters:
I20260619 13:13:51.029664 2861 leader_election.cc:288] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:13:51.029805 2866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:51.030429 2866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [term 1 LEADER]: Becoming Leader. State: Replica: 2f8dca51d3814c38b692eac3e66eff0f, State: Running, Role: LEADER
I20260619 13:13:51.030925 2866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8dca51d3814c38b692eac3e66eff0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46659 } }
I20260619 13:13:51.031507 2861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:13:51.033967 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f8dca51d3814c38b692eac3e66eff0f. Latest consensus state: current_term: 1 leader_uuid: "2f8dca51d3814c38b692eac3e66eff0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8dca51d3814c38b692eac3e66eff0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46659 } } }
I20260619 13:13:51.034294 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:51.034708 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f8dca51d3814c38b692eac3e66eff0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8dca51d3814c38b692eac3e66eff0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46659 } } }
I20260619 13:13:51.034966 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:51.038609 2875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:13:51.043653 2875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:13:51.051736 2875 catalog_manager.cc:1347] Generated new cluster ID: 6a7b841421e34923a05a9ea4fe49a685
I20260619 13:13:51.051842 2875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:13:51.066062 2875 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:13:51.067380 2875 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:13:51.088441 2875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f8dca51d3814c38b692eac3e66eff0f: Generated new TSK 0
I20260619 13:13:51.089272 2875 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:13:51.126565 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46659
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:51.431945 2887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:51.432256 2887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:51.432358 2887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:51.439719 2887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:51.439867 2887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:51.440088 2887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:13:51.449714 2887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:51.451238 2887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:51.452490 2887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:51.464653 2892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:51.464669 2893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:51.464743 2895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:51.465633 2887 server_base.cc:1034] running on GCE node
I20260619 13:13:51.466205 2887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:51.467079 2887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:51.468279 2887 hybrid_clock.cc:648] HybridClock initialized: now 1781874831468224 us; error 51 us; skew 500 ppm
I20260619 13:13:51.468560 2887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:51 dist-test-slave-mkt3 krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:51.471531 2887 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:13:51.472934 2887 webserver.cc:460] Webserver started at http://127.0.97.1:43313/ using document root <none> and password file <none>
I20260619 13:13:51.473577 2887 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:51.473690 2887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:51.474010 2887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:51.476722 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "272891d07343435dbd0ad2d3aa55428d"
format_stamp: "Formatted at 2026-06-19 13:13:51 on dist-test-slave-mkt3"
server_key: "909ee52ba105436ef4f4c1f43b0da5c5"
server_key_iv: "b3c18180a458e20823a07784b7e4488d"
server_key_version: "encryptionkey@0"
I20260619 13:13:51.477437 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "272891d07343435dbd0ad2d3aa55428d"
format_stamp: "Formatted at 2026-06-19 13:13:51 on dist-test-slave-mkt3"
server_key: "909ee52ba105436ef4f4c1f43b0da5c5"
server_key_iv: "b3c18180a458e20823a07784b7e4488d"
server_key_version: "encryptionkey@0"
I20260619 13:13:51.482203 2887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 13:13:51.485327 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:51.486593 2887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:13:51.486732 2887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "272891d07343435dbd0ad2d3aa55428d"
format_stamp: "Formatted at 2026-06-19 13:13:51 on dist-test-slave-mkt3"
server_key: "909ee52ba105436ef4f4c1f43b0da5c5"
server_key_iv: "b3c18180a458e20823a07784b7e4488d"
server_key_version: "encryptionkey@0"
I20260619 13:13:51.486900 2887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:51.507072 2887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:51.511200 2887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:51.511454 2887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:51.512334 2887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:13:51.513588 2887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:51.513664 2887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:51.513777 2887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:51.513828 2887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:51.542147 2887 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45157
I20260619 13:13:51.542200 3014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45157 every 8 connection(s)
I20260619 13:13:51.543324 2887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:13:51.552289 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 2887
I20260619 13:13:51.552495 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:13:51.553160 388 external_mini_cluster.cc:1428] Setting key bab4cf018b2f6944dedeebde11278fef
Jun 19 13:13:51 dist-test-slave-mkt3 krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:51.562337 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:51.545688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46339 (local address 127.0.97.62:46659)
0619 13:13:51.546104 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:51.546110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:51.547007 (+ 897us) server_negotiation.cc:407] Connection header received
0619 13:13:51.548728 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:51.548733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:51.548812 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:51.548905 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:51.550331 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:51.551035 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:51.552840 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:51.553242 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:51.553368 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:51.557024 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:51.557029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:51.557032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:51.557062 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:51.559390 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:51.560218 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:51.560223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:51.560227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:51.560355 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:51.560709 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:51.560714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:51.560717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:51.561053 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:51.561186 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:51.561768 (+ 582us) server_negotiation.cc:299] Negotiation successful
0619 13:13:51.562004 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":222,"threads_started":1}
I20260619 13:13:51.563118 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:51.546217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46659 (local address 127.0.97.1:46339)
0619 13:13:51.546859 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:51.546903 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:51.548478 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:51.549124 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:51.549132 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:51.549600 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:51.550172 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:51.550189 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:51.551170 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:51.551174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:51.552685 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:51.552699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:51.553467 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:51.553474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:51.553701 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:51.554337 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:51.554361 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:51.556802 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:51.559514 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:51.559523 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:51.559539 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:51.560089 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:51.560441 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:51.560444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:51.560447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:51.560615 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:51.561149 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:51.561154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:51.561441 (+ 287us) client_negotiation.cc:769] Sending connection context
0619 13:13:51.561655 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 13:13:51.562056 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":241,"threads_started":1}
I20260619 13:13:51.564478 3015 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46659
I20260619 13:13:51.564791 3015 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:51.565608 3015 heartbeater.cc:511] Master 127.0.97.62:46659 requested a full tablet report, sending...
I20260619 13:13:51.567625 2826 ts_manager.cc:194] Registered new tserver with Master: 272891d07343435dbd0ad2d3aa55428d (127.0.97.1:45157)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:13:51.569013 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:13:51.590967 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46659
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:51.893821 3021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:51.894130 3021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:51.894248 3021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:51.901314 3021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:51.901432 3021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:51.901571 3021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:13:51.911100 3021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:51.912367 3021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:51.913627 3021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:51.926441 3026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:51.926512 3027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:51.926648 3029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:51.927522 3021 server_base.cc:1034] running on GCE node
I20260619 13:13:51.928120 3021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:51.929028 3021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:51.930228 3021 hybrid_clock.cc:648] HybridClock initialized: now 1781874831930183 us; error 46 us; skew 500 ppm
I20260619 13:13:51.930507 3021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:51 dist-test-slave-mkt3 krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:51.933617 3021 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:13:51.935110 3021 webserver.cc:460] Webserver started at http://127.0.97.2:36183/ using document root <none> and password file <none>
I20260619 13:13:51.935801 3021 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:51.935915 3021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:51.936225 3021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:51.939138 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "475b122c35674e0aa35f8d6cf0b4e00e"
format_stamp: "Formatted at 2026-06-19 13:13:51 on dist-test-slave-mkt3"
server_key: "31ebec3889382299ecca00082568cf76"
server_key_iv: "055cb5ecb5124f04e8e9bb22457d540d"
server_key_version: "encryptionkey@0"
I20260619 13:13:51.939925 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "475b122c35674e0aa35f8d6cf0b4e00e"
format_stamp: "Formatted at 2026-06-19 13:13:51 on dist-test-slave-mkt3"
server_key: "31ebec3889382299ecca00082568cf76"
server_key_iv: "055cb5ecb5124f04e8e9bb22457d540d"
server_key_version: "encryptionkey@0"
I20260619 13:13:51.944979 3021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:13:51.948452 3036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:51.949740 3021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:13:51.949906 3021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "475b122c35674e0aa35f8d6cf0b4e00e"
format_stamp: "Formatted at 2026-06-19 13:13:51 on dist-test-slave-mkt3"
server_key: "31ebec3889382299ecca00082568cf76"
server_key_iv: "055cb5ecb5124f04e8e9bb22457d540d"
server_key_version: "encryptionkey@0"
I20260619 13:13:51.950084 3021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:51.980304 3021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:51.984508 3021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:51.984809 3021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:51.985754 3021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:13:51.987028 3021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:51.987104 3021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:51.987217 3021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:51.987275 3021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:52.017764 3021 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45519
I20260619 13:13:52.017814 3148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45519 every 8 connection(s)
I20260619 13:13:52.019043 3021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:13:52.027210 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 3021
I20260619 13:13:52.027379 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:13:52.027784 388 external_mini_cluster.cc:1428] Setting key 1bc1c612a31208b3c6e02a220f42e55c
Jun 19 13:13:52 dist-test-slave-mkt3 krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:52.038108 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:52.021447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56265 (local address 127.0.97.62:46659)
0619 13:13:52.021606 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:52.021610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:52.022803 (+ 1193us) server_negotiation.cc:407] Connection header received
0619 13:13:52.024602 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:52.024608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:52.024687 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:52.024783 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:52.026366 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:52.027091 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:52.028734 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:52.029155 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:52.029265 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:52.032823 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:52.032828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:52.032831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:52.032869 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:52.035267 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:52.035958 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:52.035962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:52.035965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:52.036085 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:52.036478 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:52.036481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:52.036483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:52.036728 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:52.036861 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:52.037537 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 13:13:52.037770 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:13:52.039213 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:52.021976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46659 (local address 127.0.97.2:56265)
0619 13:13:52.022641 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:52.022680 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:52.024318 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:52.025033 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:52.025042 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:52.025543 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:52.026174 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:52.026216 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:52.027249 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:52.027253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:52.028584 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:52.028597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:52.029365 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:52.029371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:52.029594 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:52.030288 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:52.030310 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:52.032612 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:52.035398 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:52.035408 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:52.035427 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:52.035838 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:52.036183 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:52.036188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:52.036192 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:52.036370 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:52.036843 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:52.036851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:52.037181 (+ 330us) client_negotiation.cc:769] Sending connection context
0619 13:13:52.037424 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 13:13:52.037996 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":257,"threads_started":1}
I20260619 13:13:52.040588 3149 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46659
I20260619 13:13:52.040930 3149 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:52.041810 3149 heartbeater.cc:511] Master 127.0.97.62:46659 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:13:52.043385 2826 ts_manager.cc:194] Registered new tserver with Master: 475b122c35674e0aa35f8d6cf0b4e00e (127.0.97.2:45519)
I20260619 13:13:52.044164 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:13:52.064414 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46659
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:52.369902 3154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:52.370189 3154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:52.370308 3154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:52.377696 3154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:52.377799 3154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:52.377970 3154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:13:52.387414 3154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46659
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:52.388626 3154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:52.390149 3154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:52.402598 3159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:52.402647 3162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:52.402621 3160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:52.403281 3154 server_base.cc:1034] running on GCE node
I20260619 13:13:52.403865 3154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:52.404757 3154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:52.405923 3154 hybrid_clock.cc:648] HybridClock initialized: now 1781874832405864 us; error 51 us; skew 500 ppm
I20260619 13:13:52.406227 3154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:52 dist-test-slave-mkt3 krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:52.409137 3154 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:13:52.410542 3154 webserver.cc:460] Webserver started at http://127.0.97.3:33255/ using document root <none> and password file <none>
I20260619 13:13:52.411173 3154 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:52.411291 3154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:52.411571 3154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:52.414357 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "859d6e030afa419bafa0cd189b1f72dd"
format_stamp: "Formatted at 2026-06-19 13:13:52 on dist-test-slave-mkt3"
server_key: "e955934439aeff09c6afce25548368bb"
server_key_iv: "913761fae1a0a8ab03046b0267d984d0"
server_key_version: "encryptionkey@0"
I20260619 13:13:52.415071 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "859d6e030afa419bafa0cd189b1f72dd"
format_stamp: "Formatted at 2026-06-19 13:13:52 on dist-test-slave-mkt3"
server_key: "e955934439aeff09c6afce25548368bb"
server_key_iv: "913761fae1a0a8ab03046b0267d984d0"
server_key_version: "encryptionkey@0"
I20260619 13:13:52.419850 3154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:13:52.423281 3169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:52.424675 3154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:13:52.424839 3154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "859d6e030afa419bafa0cd189b1f72dd"
format_stamp: "Formatted at 2026-06-19 13:13:52 on dist-test-slave-mkt3"
server_key: "e955934439aeff09c6afce25548368bb"
server_key_iv: "913761fae1a0a8ab03046b0267d984d0"
server_key_version: "encryptionkey@0"
I20260619 13:13:52.425033 3154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:52.445214 3154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:52.449594 3154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:52.449883 3154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:52.450842 3154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:13:52.452142 3154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:52.452257 3154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:52.452378 3154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:52.452428 3154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:52.481074 3154 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34927
I20260619 13:13:52.481150 3281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34927 every 8 connection(s)
I20260619 13:13:52.482307 3154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:13:52.489972 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 3154
I20260619 13:13:52.490191 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:13:52.490769 388 external_mini_cluster.cc:1428] Setting key c37fb96e1384d523ec85e40f7ea94291
Jun 19 13:13:52 dist-test-slave-mkt3 krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:52.500830 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:52.484731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34547 (local address 127.0.97.62:46659)
0619 13:13:52.484894 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:52.484898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:52.486048 (+ 1150us) server_negotiation.cc:407] Connection header received
0619 13:13:52.487795 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:52.487800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:52.487876 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:52.487962 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:52.489410 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:52.490239 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:52.492198 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:52.492655 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:52.492767 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:52.496085 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:52.496090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:52.496100 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:52.496131 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:52.498384 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:52.499002 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:52.499006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:52.499008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:52.499132 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:52.499447 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:52.499449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:52.499451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:52.499693 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:52.499792 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:52.500433 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 13:13:52.500589 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:13:52.501781 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:52.485202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46659 (local address 127.0.97.3:34547)
0619 13:13:52.485891 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:52.485936 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:52.487529 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:52.488214 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:52.488222 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:52.488708 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:52.489249 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:52.489264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:52.490396 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:52.490402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:52.492061 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:52.492074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:52.492866 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:52.492873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:52.493123 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:52.493732 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:52.493758 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:52.495876 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:52.498505 (+ 2629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:52.498512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:52.498524 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:52.498912 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:52.499209 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:52.499212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:52.499214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:52.499374 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:52.499777 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:52.499782 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:52.500080 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 13:13:52.500333 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 13:13:52.500705 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":250,"threads_started":1}
I20260619 13:13:52.503264 3282 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46659
I20260619 13:13:52.503561 3282 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:52.504312 3282 heartbeater.cc:511] Master 127.0.97.62:46659 requested a full tablet report, sending...
I20260619 13:13:52.505761 2825 ts_manager.cc:194] Registered new tserver with Master: 859d6e030afa419bafa0cd189b1f72dd (127.0.97.3:34927)
I20260619 13:13:52.506690 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34547
I20260619 13:13:52.518150 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 13:13:52.546096 3017 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:13:52.543586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.0.97.62:46659)
0619 13:13:52.543752 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:52.543756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:52.544238 (+ 482us) server_negotiation.cc:407] Connection header received
0619 13:13:52.544408 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:52.544412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:52.544477 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:52.544562 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:52.545805 (+ 1243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56598: BlockingRecv error: recv got EOF from 127.0.0.1:56598 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:13:52.549302 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 2887
W20260619 13:13:52.552031 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 2887: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:45157: server requires authentication, but client does not have Kerberos credentials available
I20260619 13:13:52.552219 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 2887
I20260619 13:13:52.564342 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 3021
W20260619 13:13:52.567026 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 3021: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:45519: server requires authentication, but client does not have Kerberos credentials available
I20260619 13:13:52.567171 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 3021
I20260619 13:13:52.578658 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 3154
W20260619 13:13:52.581489 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 3154: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:34927: server requires authentication, but client does not have Kerberos credentials available
I20260619 13:13:52.581614 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 3154
I20260619 13:13:52.593359 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 2794
I20260619 13:13:52.594535 2856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:13:52.694469 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 2794
2026-06-19T13:13:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3252 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:13:52 dist-test-slave-mkt3 krb5kdc[3299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:13:52 dist-test-slave-mkt3 krb5kdc[3299](info): set up 2 sockets
Jun 19 13:13:52 dist-test-slave-mkt3 krb5kdc[3299](info): commencing operation
krb5kdc: starting...
W20260619 13:13:53.685827 388 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:13:53 dist-test-slave-mkt3 krb5kdc[3299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:13:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:13:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:13:53.853053 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35849
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:54.155915 3312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:54.156206 3312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:54.156395 3312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:54.163646 3312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:13:54.163751 3312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:54.163806 3312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:54.163877 3312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:13:54.163936 3312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:13:54.173451 3312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:54.174715 3312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:54.175959 3312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:54.186764 3317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:54.186793 3318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:54.186813 3320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:54.187873 3312 server_base.cc:1034] running on GCE node
I20260619 13:13:54.188603 3312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:54.189921 3312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:54.191139 3312 hybrid_clock.cc:648] HybridClock initialized: now 1781874834191093 us; error 39 us; skew 500 ppm
I20260619 13:13:54.191463 3312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:54 dist-test-slave-mkt3 krb5kdc[3299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:54.194410 3312 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:13:54.195716 3312 webserver.cc:460] Webserver started at http://127.0.97.62:39267/ using document root <none> and password file <none>
I20260619 13:13:54.196341 3312 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:54.196444 3312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:54.196719 3312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:54.199579 3312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "e9664db866164e989ab05e5e476caeff"
format_stamp: "Formatted at 2026-06-19 13:13:54 on dist-test-slave-mkt3"
server_key: "c5d39e9ad75a5285726d2083360f94ad"
server_key_iv: "9c9da9f0091c4fde342df7113138a91f"
server_key_version: "encryptionkey@0"
I20260619 13:13:54.200301 3312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "e9664db866164e989ab05e5e476caeff"
format_stamp: "Formatted at 2026-06-19 13:13:54 on dist-test-slave-mkt3"
server_key: "c5d39e9ad75a5285726d2083360f94ad"
server_key_iv: "9c9da9f0091c4fde342df7113138a91f"
server_key_version: "encryptionkey@0"
I20260619 13:13:54.205138 3312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:13:54.208395 3327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:54.209779 3312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:13:54.209971 3312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "e9664db866164e989ab05e5e476caeff"
format_stamp: "Formatted at 2026-06-19 13:13:54 on dist-test-slave-mkt3"
server_key: "c5d39e9ad75a5285726d2083360f94ad"
server_key_iv: "9c9da9f0091c4fde342df7113138a91f"
server_key_version: "encryptionkey@0"
I20260619 13:13:54.210141 3312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:54.226562 3312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:54.230680 3312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:54.230940 3312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:54.247119 3312 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35849
I20260619 13:13:54.247169 3378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35849 every 8 connection(s)
I20260619 13:13:54.248410 3312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:13:54.249678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 3312
I20260619 13:13:54.249899 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:13:54.250680 388 external_mini_cluster.cc:1428] Setting key eff9b4b0fd7078af58470aa91c25be87
I20260619 13:13:54.252748 3379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:54.264775 3379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff: Bootstrap starting.
I20260619 13:13:54.267643 3379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:54.268556 3379 log.cc:824] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:13:54 dist-test-slave-mkt3 krb5kdc[3299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:54.271363 3379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff: No bootstrap required, opened a new log
I20260619 13:13:54.277311 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:54.252448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.0.97.62:35849)
0619 13:13:54.253194 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:54.253208 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:54.253254 (+ 46us) server_negotiation.cc:407] Connection header received
0619 13:13:54.255346 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:54.255371 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:54.255861 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:54.256298 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:54.257550 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:54.259521 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:54.260761 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:54.261730 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:54.261959 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:54.270565 (+ 8606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:54.270574 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:54.270590 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:54.270624 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:54.273452 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:54.274239 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:54.274257 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:54.274263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:54.274418 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:54.274793 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:54.274796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:54.274798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:54.275301 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:54.275525 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:54.275933 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 13:13:54.276306 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":316,"threads_started":1}
I20260619 13:13:54.279453 3379 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9664db866164e989ab05e5e476caeff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35849 } }
I20260619 13:13:54.279834 3379 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:54.279932 3379 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9664db866164e989ab05e5e476caeff, State: Initialized, Role: FOLLOWER
I20260619 13:13:54.280478 3379 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9664db866164e989ab05e5e476caeff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35849 } }
I20260619 13:13:54.280701 3379 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:13:54.280834 3379 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:13:54.280905 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:13:54.280995 3379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:54.283937 3379 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9664db866164e989ab05e5e476caeff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35849 } }
I20260619 13:13:54.284474 3379 leader_election.cc:302] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9664db866164e989ab05e5e476caeff; no voters:
I20260619 13:13:54.285049 3379 leader_election.cc:288] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:13:54.285135 3384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:54.286142 3384 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [term 1 LEADER]: Becoming Leader. State: Replica: e9664db866164e989ab05e5e476caeff, State: Running, Role: LEADER
I20260619 13:13:54.286640 3384 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9664db866164e989ab05e5e476caeff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35849 } }
I20260619 13:13:54.286994 3379 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:13:54.289044 3386 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9664db866164e989ab05e5e476caeff. Latest consensus state: current_term: 1 leader_uuid: "e9664db866164e989ab05e5e476caeff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9664db866164e989ab05e5e476caeff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35849 } } }
I20260619 13:13:54.289393 3386 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:54.289970 3385 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9664db866164e989ab05e5e476caeff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9664db866164e989ab05e5e476caeff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35849 } } }
I20260619 13:13:54.290290 3385 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:54.290424 3392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:13:54.305619 3392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:13:54.314827 3392 catalog_manager.cc:1347] Generated new cluster ID: e82e911faf194bedb8a82b0c9f23bcc0
I20260619 13:13:54.314934 3392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:13:54.339354 3392 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:13:54.341270 3392 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:13:54.346607 3408 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 13:13:54.350224 3392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9664db866164e989ab05e5e476caeff: Generated new TSK 0
I20260619 13:13:54.351194 3392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:13:54.354460 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:54.340166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.0.97.62:35849)
0619 13:13:54.340489 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:54.340495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:54.347646 (+ 7151us) server_negotiation.cc:407] Connection header received
0619 13:13:54.347831 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:54.347836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:54.347922 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:54.348042 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:54.349267 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:54.350280 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:54.351596 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:54.352034 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:54.352146 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:54.353139 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:54.353146 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:54.353149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:13:54.353739 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:54.353783 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:13:54.353810 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:54.353937 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:54.354053 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 13:13:54.354151 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:13:54.361725 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 3312
I20260619 13:13:54.362741 3374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:13:54.447953 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 3312
2026-06-19T13:13:54Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1748 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:13:54 dist-test-slave-mkt3 krb5kdc[3413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:13:54 dist-test-slave-mkt3 krb5kdc[3413](info): set up 2 sockets
Jun 19 13:13:54 dist-test-slave-mkt3 krb5kdc[3413](info): commencing operation
krb5kdc: starting...
W20260619 13:13:55.437824 388 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:13:55 dist-test-slave-mkt3 krb5kdc[3413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:13:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:13:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:13:55.601179 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33085
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:55.901796 3426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:55.902132 3426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:55.902266 3426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:55.909307 3426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:13:55.909407 3426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:55.909477 3426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:55.909546 3426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:13:55.909606 3426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:13:55.918995 3426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:55.920226 3426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:55.921446 3426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:55.932152 3434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:55.932236 3431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:55.932236 3432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:55.933105 3426 server_base.cc:1034] running on GCE node
I20260619 13:13:55.933786 3426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:55.935082 3426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:55.936268 3426 hybrid_clock.cc:648] HybridClock initialized: now 1781874835936221 us; error 40 us; skew 500 ppm
I20260619 13:13:55.936559 3426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:55 dist-test-slave-mkt3 krb5kdc[3413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:55.939320 3426 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:13:55.940553 3426 webserver.cc:460] Webserver started at http://127.0.97.62:46097/ using document root <none> and password file <none>
I20260619 13:13:55.941185 3426 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:55.941298 3426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:55.941565 3426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:55.944296 3426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "69411875b4444b1999ea24a1d5590b1c"
format_stamp: "Formatted at 2026-06-19 13:13:55 on dist-test-slave-mkt3"
server_key: "87e76c52a53e70ede4e8eb61ee043618"
server_key_iv: "51369a48c2b73a0b74c952bd14289b3c"
server_key_version: "encryptionkey@0"
I20260619 13:13:55.944983 3426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "69411875b4444b1999ea24a1d5590b1c"
format_stamp: "Formatted at 2026-06-19 13:13:55 on dist-test-slave-mkt3"
server_key: "87e76c52a53e70ede4e8eb61ee043618"
server_key_iv: "51369a48c2b73a0b74c952bd14289b3c"
server_key_version: "encryptionkey@0"
I20260619 13:13:55.949671 3426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 13:13:55.952718 3441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:55.953903 3426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 13:13:55.954046 3426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "69411875b4444b1999ea24a1d5590b1c"
format_stamp: "Formatted at 2026-06-19 13:13:55 on dist-test-slave-mkt3"
server_key: "87e76c52a53e70ede4e8eb61ee043618"
server_key_iv: "51369a48c2b73a0b74c952bd14289b3c"
server_key_version: "encryptionkey@0"
I20260619 13:13:55.954210 3426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:55.992856 3426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:55.996623 3426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:55.996879 3426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:56.013280 3426 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33085
I20260619 13:13:56.013309 3492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33085 every 8 connection(s)
I20260619 13:13:56.014578 3426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:13:56.015394 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 3426
I20260619 13:13:56.015599 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:13:56.016201 388 external_mini_cluster.cc:1428] Setting key adcd46788f145ac7cec2c14bc42e1c32
I20260619 13:13:56.018690 3493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 13:13:56 dist-test-slave-mkt3 krb5kdc[3413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:56.028395 3493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c: Bootstrap starting.
I20260619 13:13:56.031490 3493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:56.032508 3493 log.cc:824] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:56.035538 3493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c: No bootstrap required, opened a new log
I20260619 13:13:56.035912 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:56.018072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46318 (local address 127.0.97.62:33085)
0619 13:13:56.018779 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:56.018795 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:56.018839 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:13:56.020329 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:56.020349 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:56.020694 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:56.021077 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:56.022367 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:56.023633 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:56.024800 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:56.025412 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:56.025579 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:56.029297 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:56.029304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:56.029317 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:56.029353 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:56.032120 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:56.032851 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:56.032858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:56.032863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:56.033054 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:56.033403 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:56.033407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:56.033411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:56.034011 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:56.034229 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:56.034615 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 13:13:56.034923 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":289,"threads_started":1}
I20260619 13:13:56.042896 3493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69411875b4444b1999ea24a1d5590b1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33085 } }
I20260619 13:13:56.043210 3493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:56.043305 3493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69411875b4444b1999ea24a1d5590b1c, State: Initialized, Role: FOLLOWER
I20260619 13:13:56.043787 3493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69411875b4444b1999ea24a1d5590b1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33085 } }
I20260619 13:13:56.043987 3493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:13:56.044106 3493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:13:56.044265 3493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:56.046480 3493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69411875b4444b1999ea24a1d5590b1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33085 } }
I20260619 13:13:56.046902 3493 leader_election.cc:302] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69411875b4444b1999ea24a1d5590b1c; no voters:
I20260619 13:13:56.047449 3493 leader_election.cc:288] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:13:56.047528 3498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:56.048295 3498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [term 1 LEADER]: Becoming Leader. State: Replica: 69411875b4444b1999ea24a1d5590b1c, State: Running, Role: LEADER
I20260619 13:13:56.048759 3498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69411875b4444b1999ea24a1d5590b1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33085 } }
I20260619 13:13:56.049458 3493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:13:56.050994 3500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69411875b4444b1999ea24a1d5590b1c. Latest consensus state: current_term: 1 leader_uuid: "69411875b4444b1999ea24a1d5590b1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69411875b4444b1999ea24a1d5590b1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33085 } } }
I20260619 13:13:56.051314 3500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:56.051729 3499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69411875b4444b1999ea24a1d5590b1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69411875b4444b1999ea24a1d5590b1c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33085 } } }
I20260619 13:13:56.051990 3499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:56.053187 3504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:13:56.058137 3504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:13:56.066200 3504 catalog_manager.cc:1347] Generated new cluster ID: 66ac3e37d1f04c7caaf70d6a5b28f7a0
I20260619 13:13:56.066320 3504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:13:56.097649 3504 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:13:56.099020 3504 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:13:56.108117 3504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69411875b4444b1999ea24a1d5590b1c: Generated new TSK 0
I20260619 13:13:56.108942 3504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:13:56.149309 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33085
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:56.457928 3520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:56.458222 3520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:56.458360 3520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:56.465400 3520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:56.465513 3520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:56.465662 3520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:13:56.474830 3520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:56.475976 3520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:56.477170 3520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:56.488948 3525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:56.488910 3526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:56.488978 3528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:56.489670 3520 server_base.cc:1034] running on GCE node
I20260619 13:13:56.490252 3520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:56.491117 3520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:56.492316 3520 hybrid_clock.cc:648] HybridClock initialized: now 1781874836492265 us; error 44 us; skew 500 ppm
I20260619 13:13:56.492594 3520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:56 dist-test-slave-mkt3 krb5kdc[3413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:56.495510 3520 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:13:56.496820 3520 webserver.cc:460] Webserver started at http://127.0.97.1:41295/ using document root <none> and password file <none>
I20260619 13:13:56.497437 3520 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:56.497537 3520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:56.497810 3520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:56.500597 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "69538e081557401dbcad1b287651549f"
format_stamp: "Formatted at 2026-06-19 13:13:56 on dist-test-slave-mkt3"
server_key: "cb320f61c049168ba2a94403029d7359"
server_key_iv: "ac1000e51ada16240998a7eab97b7007"
server_key_version: "encryptionkey@0"
I20260619 13:13:56.501581 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "69538e081557401dbcad1b287651549f"
format_stamp: "Formatted at 2026-06-19 13:13:56 on dist-test-slave-mkt3"
server_key: "cb320f61c049168ba2a94403029d7359"
server_key_iv: "ac1000e51ada16240998a7eab97b7007"
server_key_version: "encryptionkey@0"
I20260619 13:13:56.506515 3520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:13:56.509765 3535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:56.511148 3520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:13:56.511298 3520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "69538e081557401dbcad1b287651549f"
format_stamp: "Formatted at 2026-06-19 13:13:56 on dist-test-slave-mkt3"
server_key: "cb320f61c049168ba2a94403029d7359"
server_key_iv: "ac1000e51ada16240998a7eab97b7007"
server_key_version: "encryptionkey@0"
I20260619 13:13:56.511467 3520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:56.544835 3520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:56.548465 3520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:56.548707 3520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:56.549571 3520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:13:56.550843 3520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:56.550916 3520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:56.551033 3520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:56.551082 3520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:56.579066 3520 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34811
I20260619 13:13:56.579124 3647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34811 every 8 connection(s)
I20260619 13:13:56.580456 3520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:13:56.589125 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 3520
I20260619 13:13:56.589283 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:13:56.589888 388 external_mini_cluster.cc:1428] Setting key e118254bea633ca188836e2928b75973
Jun 19 13:13:56 dist-test-slave-mkt3 krb5kdc[3413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:56.598397 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:56.582833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43381 (local address 127.0.97.62:33085)
0619 13:13:56.583236 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:56.583241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:56.584176 (+ 935us) server_negotiation.cc:407] Connection header received
0619 13:13:56.585879 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:56.585884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:56.585960 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:56.586052 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:56.587449 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:56.588152 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:56.589662 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:56.590122 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:56.590241 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:56.593348 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:56.593355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:56.593358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:56.593387 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:56.595595 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:56.596216 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:56.596228 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:56.596232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:56.596339 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:56.596762 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:56.596766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:56.596768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:56.597037 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:56.597152 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:56.597952 (+ 800us) server_negotiation.cc:299] Negotiation successful
0619 13:13:56.598137 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":201,"threads_started":1}
I20260619 13:13:56.599581 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:56.583402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33085 (local address 127.0.97.1:43381)
0619 13:13:56.584030 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:56.584069 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:56.585606 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:56.586280 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:56.586289 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:56.586761 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:56.587290 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:56.587305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:56.588290 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:56.588294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:56.589535 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:56.589546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:56.590326 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:56.590333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:56.590541 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:56.591121 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:56.591153 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:56.593142 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:56.595715 (+ 2573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:56.595731 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:56.595746 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:56.596114 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:56.596441 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:56.596446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:56.596450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:56.596617 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:56.597157 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:56.597166 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:56.597542 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 13:13:56.597824 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 13:13:56.598328 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":250,"threads_started":1}
I20260619 13:13:56.600972 3648 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33085
I20260619 13:13:56.601277 3648 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:56.602108 3648 heartbeater.cc:511] Master 127.0.97.62:33085 requested a full tablet report, sending...
I20260619 13:13:56.604147 3458 ts_manager.cc:194] Registered new tserver with Master: 69538e081557401dbcad1b287651549f (127.0.97.1:34811)
I20260619 13:13:56.605513 3458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43381
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:13:56.626106 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33085
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:56.921860 3654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:56.922163 3654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:56.922298 3654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:56.929405 3654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:56.929493 3654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:56.929631 3654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:13:56.938967 3654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:56.940121 3654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:56.941351 3654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:56.954918 3659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:56.955044 3662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:56.955067 3660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:56.955991 3654 server_base.cc:1034] running on GCE node
I20260619 13:13:56.956537 3654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:56.957423 3654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:56.958602 3654 hybrid_clock.cc:648] HybridClock initialized: now 1781874836958548 us; error 44 us; skew 500 ppm
I20260619 13:13:56.958899 3654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:56 dist-test-slave-mkt3 krb5kdc[3413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:56.961607 3654 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:13:56.962924 3654 webserver.cc:460] Webserver started at http://127.0.97.2:46045/ using document root <none> and password file <none>
I20260619 13:13:56.963552 3654 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:56.963660 3654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:56.963918 3654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:56.966660 3654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "e3774f5546a049f6a261525f0ad35cfc"
format_stamp: "Formatted at 2026-06-19 13:13:56 on dist-test-slave-mkt3"
server_key: "930bd741b22b383e535dceb336f9b1a2"
server_key_iv: "2328bd8a7b6c0d4b2a2bfd739ffabe8b"
server_key_version: "encryptionkey@0"
I20260619 13:13:56.967363 3654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e3774f5546a049f6a261525f0ad35cfc"
format_stamp: "Formatted at 2026-06-19 13:13:56 on dist-test-slave-mkt3"
server_key: "930bd741b22b383e535dceb336f9b1a2"
server_key_iv: "2328bd8a7b6c0d4b2a2bfd739ffabe8b"
server_key_version: "encryptionkey@0"
I20260619 13:13:56.971925 3654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 13:13:56.975054 3669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:56.976275 3654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 13:13:56.976408 3654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "e3774f5546a049f6a261525f0ad35cfc"
format_stamp: "Formatted at 2026-06-19 13:13:56 on dist-test-slave-mkt3"
server_key: "930bd741b22b383e535dceb336f9b1a2"
server_key_iv: "2328bd8a7b6c0d4b2a2bfd739ffabe8b"
server_key_version: "encryptionkey@0"
I20260619 13:13:56.976564 3654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:56.988234 3654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:56.991786 3654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:56.992033 3654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:56.992868 3654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:13:56.994148 3654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:56.994230 3654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:56.994340 3654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:56.994391 3654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:57.026137 3654 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33035
I20260619 13:13:57.026144 3781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33035 every 8 connection(s)
I20260619 13:13:57.027700 3654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:13:57.030225 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 3654
I20260619 13:13:57.030398 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:13:57.030906 388 external_mini_cluster.cc:1428] Setting key b921fd6b980112147977e4991cd39b88
Jun 19 13:13:57 dist-test-slave-mkt3 krb5kdc[3413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:13:57.047247 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:57.031605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51489 (local address 127.0.97.62:33085)
0619 13:13:57.031771 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:57.031777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:57.031923 (+ 146us) server_negotiation.cc:407] Connection header received
0619 13:13:57.034113 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:57.034118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:57.034208 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:57.034303 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:57.035643 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.036596 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.038058 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.038458 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.038562 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:57.041898 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:57.041903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:57.041907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:57.041936 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:57.044269 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:57.045078 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:57.045081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:57.045084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:57.045191 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:57.045621 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:57.045624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:57.045626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:57.045881 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:57.045983 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:57.046814 (+ 831us) server_negotiation.cc:299] Negotiation successful
0619 13:13:57.046975 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:13:57.048302 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:57.030954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33085 (local address 127.0.97.2:51489)
0619 13:13:57.031775 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:57.031819 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:57.033830 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:57.034493 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:57.034501 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:57.034963 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:57.035497 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:57.035513 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.036713 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.036717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:57.037932 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:57.037945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.038656 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.038663 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:57.038884 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:57.039471 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:57.039493 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:57.041683 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:57.044401 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:57.044409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:57.044425 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:57.044974 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:57.045283 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:57.045288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:57.045292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:57.045531 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:57.045990 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:57.045998 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:57.046442 (+ 444us) client_negotiation.cc:769] Sending connection context
0619 13:13:57.046747 (+ 305us) client_negotiation.cc:240] Negotiation successful
0619 13:13:57.047265 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":302,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 13:13:57.049651 3782 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33085
I20260619 13:13:57.049963 3782 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:57.050727 3782 heartbeater.cc:511] Master 127.0.97.62:33085 requested a full tablet report, sending...
I20260619 13:13:57.052124 3458 ts_manager.cc:194] Registered new tserver with Master: e3774f5546a049f6a261525f0ad35cfc (127.0.97.2:33035)
I20260619 13:13:57.052902 3458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:13:57.068626 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33085
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:57.367524 3787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:57.367805 3787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:57.367941 3787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:57.374967 3787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:57.375077 3787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:57.375229 3787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:13:57.384845 3787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:57.386073 3787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:57.387291 3787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:57.398733 3793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:57.398754 3792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:57.399269 3795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:57.399339 3787 server_base.cc:1034] running on GCE node
I20260619 13:13:57.399938 3787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:57.400787 3787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:57.401938 3787 hybrid_clock.cc:648] HybridClock initialized: now 1781874837401868 us; error 61 us; skew 500 ppm
I20260619 13:13:57.402222 3787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:57 dist-test-slave-mkt3 krb5kdc[3413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:57.405123 3787 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:13:57.406401 3787 webserver.cc:460] Webserver started at http://127.0.97.3:44953/ using document root <none> and password file <none>
I20260619 13:13:57.407025 3787 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:57.407130 3787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:57.407382 3787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:57.410058 3787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "c7b36c01492e447b8209e7234ccd078a"
format_stamp: "Formatted at 2026-06-19 13:13:57 on dist-test-slave-mkt3"
server_key: "ca1627a596a6cb95e3399a5db2f2475c"
server_key_iv: "e99b6f0df5a1d3fde5cca4eb7adb0efa"
server_key_version: "encryptionkey@0"
I20260619 13:13:57.410753 3787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c7b36c01492e447b8209e7234ccd078a"
format_stamp: "Formatted at 2026-06-19 13:13:57 on dist-test-slave-mkt3"
server_key: "ca1627a596a6cb95e3399a5db2f2475c"
server_key_iv: "e99b6f0df5a1d3fde5cca4eb7adb0efa"
server_key_version: "encryptionkey@0"
I20260619 13:13:57.415364 3787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:13:57.418551 3802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:57.419809 3787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:13:57.419945 3787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "c7b36c01492e447b8209e7234ccd078a"
format_stamp: "Formatted at 2026-06-19 13:13:57 on dist-test-slave-mkt3"
server_key: "ca1627a596a6cb95e3399a5db2f2475c"
server_key_iv: "e99b6f0df5a1d3fde5cca4eb7adb0efa"
server_key_version: "encryptionkey@0"
I20260619 13:13:57.420100 3787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:57.454757 3787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:57.458494 3787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:57.458746 3787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:57.459739 3787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:13:57.460952 3787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:57.461036 3787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:57.461158 3787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:57.461208 3787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:57.489545 3787 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42753
I20260619 13:13:57.489591 3914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42753 every 8 connection(s)
I20260619 13:13:57.490746 3787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:13:57.492686 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 3787
I20260619 13:13:57.492830 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:13:57.493324 388 external_mini_cluster.cc:1428] Setting key e03c0d8fbc8ce1bfc913b07798d86d76
Jun 19 13:13:57 dist-test-slave-mkt3 krb5kdc[3413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:57.509258 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:57.493829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59843 (local address 127.0.97.62:33085)
0619 13:13:57.493992 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:57.493998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:57.494667 (+ 669us) server_negotiation.cc:407] Connection header received
0619 13:13:57.496401 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:57.496406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:57.496464 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:57.496540 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:57.497981 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.498671 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.500187 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.500561 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.500646 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:57.504137 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:57.504142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:57.504145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:57.504177 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:57.506466 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:57.507192 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:57.507197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:57.507200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:57.507339 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:57.507752 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:57.507756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:57.507759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:57.508086 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:57.508189 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:57.508857 (+ 668us) server_negotiation.cc:299] Negotiation successful
0619 13:13:57.509007 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 13:13:57.510247 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:57.493663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33085 (local address 127.0.97.3:59843)
0619 13:13:57.494510 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:57.494556 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:57.496174 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:57.496782 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:57.496790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:57.497273 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:57.497816 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:57.497833 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.498826 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.498830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:57.500081 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:57.500093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.500760 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.500765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:57.500962 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:57.501625 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:57.501651 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:57.503944 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:57.506603 (+ 2659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:57.506611 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:57.506624 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:57.507072 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:57.507440 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:57.507444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:57.507448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:57.507658 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:57.508211 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:57.508222 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:57.508541 (+ 319us) client_negotiation.cc:769] Sending connection context
0619 13:13:57.508779 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 13:13:57.509152 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":382,"threads_started":1}
I20260619 13:13:57.511492 3915 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33085
I20260619 13:13:57.511739 3915 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:57.512362 3915 heartbeater.cc:511] Master 127.0.97.62:33085 requested a full tablet report, sending...
I20260619 13:13:57.513622 3458 ts_manager.cc:194] Registered new tserver with Master: c7b36c01492e447b8209e7234ccd078a (127.0.97.3:42753)
I20260619 13:13:57.514391 3458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59843
I20260619 13:13:57.520049 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:13:57 dist-test-slave-mkt3 krb5kdc[3413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874837, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 13:13:57 dist-test-slave-mkt3 krb5kdc[3413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874837, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:57.561471 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:57.549040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.0.97.62:33085)
0619 13:13:57.549262 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:57.549268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:57.549559 (+ 291us) server_negotiation.cc:407] Connection header received
0619 13:13:57.549704 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:57.549708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:57.549780 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:57.549890 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:57.551109 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.551808 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.552971 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.553341 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.553428 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:57.556909 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:57.556914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:57.556917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:57.556953 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:57.559377 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:57.559918 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:57.559922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:57.559925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:57.560070 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:57.560389 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:57.560391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:57.560393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:57.560671 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:57.560854 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:57.561103 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 13:13:57.561251 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 13:13:57.561981 3458 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46334
I20260619 13:13:57.563513 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 3520
Jun 19 13:13:57 dist-test-slave-mkt3 krb5kdc[3413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874837, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:13:57.576776 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:57.564364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.0.97.1:34811)
0619 13:13:57.564864 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:57.564871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:57.564893 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:13:57.564942 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:57.564948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:57.565139 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:57.565259 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:57.566292 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.567219 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.568325 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.568830 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.568917 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:57.572298 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:57.572308 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:57.572321 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:57.572352 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:57.574693 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:57.575234 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:57.575240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:57.575242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:57.575342 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:57.575592 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:57.575595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:57.575597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:57.575880 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:57.576074 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:57.576304 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 13:13:57.576475 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":254,"threads_started":1}
W20260619 13:13:57.577942 3643 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59426
W20260619 13:13:57.579000 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 3520: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:13:57.579206 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 3520
I20260619 13:13:57.591117 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 3654
Jun 19 13:13:57 dist-test-slave-mkt3 krb5kdc[3413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874837, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:13:57.605397 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:57.591874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.0.97.2:33035)
0619 13:13:57.592364 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:57.592370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:57.592394 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:13:57.592445 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:57.592451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:57.592638 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:57.592761 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:57.593827 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.594852 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.595970 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.596462 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.596549 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:57.600107 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:57.600117 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:57.600129 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:57.600157 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:57.602690 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:57.603284 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:57.603291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:57.603293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:57.603405 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:57.603750 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:57.603754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:57.603756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:57.604290 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:57.604555 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:57.604848 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 13:13:57.605061 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":238,"threads_started":1}
W20260619 13:13:57.606698 3777 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56272
W20260619 13:13:57.607740 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 3654: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:13:57.607867 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 3654
I20260619 13:13:57.619287 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 3787
Jun 19 13:13:57 dist-test-slave-mkt3 krb5kdc[3413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874837, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:13:57.633390 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:57.620046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.0.97.3:42753)
0619 13:13:57.620609 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:57.620615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:57.620643 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:13:57.620696 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:57.620702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:57.620895 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:57.621033 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:57.622161 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.623176 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.624366 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:57.624983 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:57.625075 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:57.628782 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:57.628792 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:57.628805 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:57.628834 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:57.631243 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:57.631777 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:57.631783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:57.631785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:57.631889 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:57.632131 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:57.632134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:57.632136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:57.632435 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:57.632651 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:57.632857 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 13:13:57.633038 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":273,"threads_started":1}
W20260619 13:13:57.634671 3910 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40926
W20260619 13:13:57.635663 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 3787: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:13:57.635807 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 3787
I20260619 13:13:57.647306 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 3426
I20260619 13:13:57.648272 3488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:13:57.746039 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 3426
2026-06-19T13:13:57Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3304 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:13:57 dist-test-slave-mkt3 krb5kdc[3932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:13:57 dist-test-slave-mkt3 krb5kdc[3932](info): set up 2 sockets
Jun 19 13:13:57 dist-test-slave-mkt3 krb5kdc[3932](info): commencing operation
krb5kdc: starting...
W20260619 13:13:58.745873 388 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:13:58 dist-test-slave-mkt3 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:13:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:13:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:13:58.917544 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46853
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:59.230783 3945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:59.231097 3945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:59.231225 3945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:59.238365 3945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:13:59.238477 3945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:59.238551 3945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:59.238622 3945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:13:59.238679 3945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:13:59.248453 3945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:59.249706 3945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:59.251019 3945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:59.262042 3950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:59.261926 3951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:59.262070 3953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:59.263159 3945 server_base.cc:1034] running on GCE node
I20260619 13:13:59.263929 3945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:59.265296 3945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:59.266516 3945 hybrid_clock.cc:648] HybridClock initialized: now 1781874839266452 us; error 51 us; skew 500 ppm
I20260619 13:13:59.266839 3945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:59 dist-test-slave-mkt3 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:59.273762 3945 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:13:59.275267 3945 webserver.cc:460] Webserver started at http://127.0.97.62:39455/ using document root <none> and password file <none>
I20260619 13:13:59.275914 3945 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:59.276027 3945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:59.276316 3945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:59.279177 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "a45419b1a290470093809f429978c39e"
format_stamp: "Formatted at 2026-06-19 13:13:59 on dist-test-slave-mkt3"
server_key: "2a947627a60aff695b396d8607ff86a1"
server_key_iv: "a3733bd59de1b5b05e0ea7873ecb68b6"
server_key_version: "encryptionkey@0"
I20260619 13:13:59.279907 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "a45419b1a290470093809f429978c39e"
format_stamp: "Formatted at 2026-06-19 13:13:59 on dist-test-slave-mkt3"
server_key: "2a947627a60aff695b396d8607ff86a1"
server_key_iv: "a3733bd59de1b5b05e0ea7873ecb68b6"
server_key_version: "encryptionkey@0"
I20260619 13:13:59.284883 3945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:13:59.288108 3960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:59.289345 3945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:13:59.289494 3945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "a45419b1a290470093809f429978c39e"
format_stamp: "Formatted at 2026-06-19 13:13:59 on dist-test-slave-mkt3"
server_key: "2a947627a60aff695b396d8607ff86a1"
server_key_iv: "a3733bd59de1b5b05e0ea7873ecb68b6"
server_key_version: "encryptionkey@0"
I20260619 13:13:59.289659 3945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:59.304316 3945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:59.308302 3945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:59.308545 3945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:59.324654 3945 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46853
I20260619 13:13:59.324683 4011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46853 every 8 connection(s)
I20260619 13:13:59.325968 3945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:13:59.329582 4012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:13:59.335436 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 3945
I20260619 13:13:59.335654 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:13:59.336241 388 external_mini_cluster.cc:1428] Setting key 00be5c0d8c20d543711347ac2dd5ac8b
I20260619 13:13:59.339597 4012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e: Bootstrap starting.
I20260619 13:13:59.343068 4012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e: Neither blocks nor log segments found. Creating new log.
I20260619 13:13:59.344069 4012 log.cc:824] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:13:59.346865 4012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e: No bootstrap required, opened a new log
Jun 19 13:13:59 dist-test-slave-mkt3 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:59.355809 4012 raft_consensus.cc:348] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45419b1a290470093809f429978c39e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46853 } }
I20260619 13:13:59.356240 4012 raft_consensus.cc:374] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:13:59.356331 4012 raft_consensus.cc:729] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a45419b1a290470093809f429978c39e, State: Initialized, Role: FOLLOWER
I20260619 13:13:59.356940 4012 consensus_queue.cc:260] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45419b1a290470093809f429978c39e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46853 } }
I20260619 13:13:59.357169 4012 raft_consensus.cc:388] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:13:59.357326 4012 raft_consensus.cc:482] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:13:59.357497 4012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:13:59.358834 4015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:59.338084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.0.97.62:46853)
0619 13:13:59.338761 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:59.338773 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:59.338822 (+ 49us) server_negotiation.cc:407] Connection header received
0619 13:13:59.340499 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:59.340522 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:59.340895 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:59.341250 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:59.343051 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:59.344377 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:59.345951 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:59.346610 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:59.346741 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:59.351304 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:59.351312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:59.351329 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:59.351367 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:59.354548 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:59.355371 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:59.355378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:59.355385 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:59.355546 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:59.355925 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:59.355928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:59.355930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:59.356553 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:59.356841 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:59.357303 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 13:13:59.357734 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":303,"threads_started":1}
I20260619 13:13:59.359839 4012 raft_consensus.cc:504] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45419b1a290470093809f429978c39e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46853 } }
I20260619 13:13:59.360312 4012 leader_election.cc:302] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a45419b1a290470093809f429978c39e; no voters:
I20260619 13:13:59.360872 4012 leader_election.cc:288] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:13:59.361001 4017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:13:59.362398 4017 raft_consensus.cc:686] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [term 1 LEADER]: Becoming Leader. State: Replica: a45419b1a290470093809f429978c39e, State: Running, Role: LEADER
I20260619 13:13:59.362804 4012 sys_catalog.cc:564] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:13:59.362846 4017 consensus_queue.cc:237] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45419b1a290470093809f429978c39e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46853 } }
I20260619 13:13:59.365487 4019 sys_catalog.cc:455] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a45419b1a290470093809f429978c39e. Latest consensus state: current_term: 1 leader_uuid: "a45419b1a290470093809f429978c39e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45419b1a290470093809f429978c39e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46853 } } }
I20260619 13:13:59.365880 4019 sys_catalog.cc:458] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:59.366101 4018 sys_catalog.cc:455] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a45419b1a290470093809f429978c39e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45419b1a290470093809f429978c39e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46853 } } }
I20260619 13:13:59.366389 4018 sys_catalog.cc:458] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e [sys.catalog]: This master's current role is: LEADER
I20260619 13:13:59.367839 4024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:13:59.377146 4024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:13:59.385010 4024 catalog_manager.cc:1347] Generated new cluster ID: 9f55e57dd39b4fafaee26d2e3343ec12
I20260619 13:13:59.385123 4024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:13:59.397658 4024 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:13:59.398945 4024 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:13:59.406014 4024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a45419b1a290470093809f429978c39e: Generated new TSK 0
I20260619 13:13:59.406822 4024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:13:59.454552 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46853
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:13:59.759294 4038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:13:59.759542 4038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:13:59.759630 4038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:13:59.766523 4038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:13:59.766615 4038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:13:59.766728 4038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:13:59.775832 4038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:13:59.776950 4038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:13:59.778193 4038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:13:59.790488 4043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:59.790446 4044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:13:59.790628 4046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:13:59.791510 4038 server_base.cc:1034] running on GCE node
I20260619 13:13:59.792074 4038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:13:59.792930 4038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:13:59.794112 4038 hybrid_clock.cc:648] HybridClock initialized: now 1781874839794059 us; error 41 us; skew 500 ppm
I20260619 13:13:59.794389 4038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:13:59 dist-test-slave-mkt3 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:13:59.797246 4038 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:13:59.798576 4038 webserver.cc:460] Webserver started at http://127.0.97.1:33079/ using document root <none> and password file <none>
I20260619 13:13:59.799189 4038 fs_manager.cc:359] Metadata directory not provided
I20260619 13:13:59.799300 4038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:13:59.799578 4038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:13:59.802316 4038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "da397c9a7a9c4c1faf8287a2b7205ae8"
format_stamp: "Formatted at 2026-06-19 13:13:59 on dist-test-slave-mkt3"
server_key: "1fad82f7b95214af6e53f3094153f1c3"
server_key_iv: "082efd96fc16cf87a5c24cc63f29e0de"
server_key_version: "encryptionkey@0"
I20260619 13:13:59.803025 4038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "da397c9a7a9c4c1faf8287a2b7205ae8"
format_stamp: "Formatted at 2026-06-19 13:13:59 on dist-test-slave-mkt3"
server_key: "1fad82f7b95214af6e53f3094153f1c3"
server_key_iv: "082efd96fc16cf87a5c24cc63f29e0de"
server_key_version: "encryptionkey@0"
I20260619 13:13:59.807724 4038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:13:59.810895 4053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:59.812127 4038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260619 13:13:59.812259 4038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "da397c9a7a9c4c1faf8287a2b7205ae8"
format_stamp: "Formatted at 2026-06-19 13:13:59 on dist-test-slave-mkt3"
server_key: "1fad82f7b95214af6e53f3094153f1c3"
server_key_iv: "082efd96fc16cf87a5c24cc63f29e0de"
server_key_version: "encryptionkey@0"
I20260619 13:13:59.812422 4038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:13:59.833292 4038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:13:59.837222 4038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:13:59.837461 4038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:13:59.838352 4038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:13:59.839524 4038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:13:59.839610 4038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:59.839723 4038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:13:59.839776 4038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:13:59.866988 4038 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40173
I20260619 13:13:59.867022 4165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40173 every 8 connection(s)
I20260619 13:13:59.868251 4038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:13:59.870918 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 4038
I20260619 13:13:59.871120 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:13:59.871701 388 external_mini_cluster.cc:1428] Setting key 3587a8dd93783e854479d9236b79dbe9
Jun 19 13:13:59 dist-test-slave-mkt3 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:13:59.888036 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:59.870556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52863 (local address 127.0.97.62:46853)
0619 13:13:59.870968 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 13:13:59.870974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:13:59.872111 (+ 1137us) server_negotiation.cc:407] Connection header received
0619 13:13:59.874088 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:13:59.874094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:13:59.874173 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:13:59.874288 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:13:59.875684 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:59.876627 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:59.878247 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:13:59.878760 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:13:59.878880 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:59.882746 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:13:59.882751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:13:59.882754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:13:59.882785 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:13:59.885193 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:59.885914 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:59.885918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:59.885920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:59.886031 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:13:59.886419 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:13:59.886422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:13:59.886424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:13:59.886691 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:13:59.886814 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 13:13:59.887473 (+ 659us) server_negotiation.cc:299] Negotiation successful
0619 13:13:59.887694 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":194,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:13:59.888954 4169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:13:59.871190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46853 (local address 127.0.97.1:52863)
0619 13:13:59.871952 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0619 13:13:59.871996 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:13:59.873802 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:13:59.874461 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:13:59.874469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:13:59.874984 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:13:59.875529 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:59.875548 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:59.876773 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:59.876778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:59.878104 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:13:59.878116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:13:59.878968 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:13:59.878975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:13:59.879260 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:13:59.879989 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:13:59.880016 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:13:59.882528 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:13:59.885318 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:59.885327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:59.885345 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:59.885773 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:59.886128 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:13:59.886134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:13:59.886138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:13:59.886312 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:13:59.886805 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:13:59.886813 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:13:59.887133 (+ 320us) client_negotiation.cc:769] Sending connection context
0619 13:13:59.887357 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 13:13:59.887852 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":298,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:13:59.890450 4166 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46853
I20260619 13:13:59.890763 4166 heartbeater.cc:464] Registering TS with master...
I20260619 13:13:59.891417 4166 heartbeater.cc:511] Master 127.0.97.62:46853 requested a full tablet report, sending...
I20260619 13:13:59.893549 3977 ts_manager.cc:194] Registered new tserver with Master: da397c9a7a9c4c1faf8287a2b7205ae8 (127.0.97.1:40173)
I20260619 13:13:59.895193 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:13:59.911927 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46853
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:00.208441 4172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:00.208742 4172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:00.208889 4172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:00.215993 4172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:00.216120 4172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:00.216275 4172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:14:00.225636 4172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:00.226853 4172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:00.228071 4172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:00.239971 4177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:00.239938 4178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:00.240006 4180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:00.241082 4172 server_base.cc:1034] running on GCE node
I20260619 13:14:00.241621 4172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:00.242529 4172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:00.243713 4172 hybrid_clock.cc:648] HybridClock initialized: now 1781874840243663 us; error 43 us; skew 500 ppm
I20260619 13:14:00.243996 4172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:00 dist-test-slave-mkt3 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:00.246800 4172 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:14:00.248104 4172 webserver.cc:460] Webserver started at http://127.0.97.2:45077/ using document root <none> and password file <none>
I20260619 13:14:00.248714 4172 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:00.248823 4172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:00.249091 4172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:00.251784 4172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "ebfaaf81a5af4a129f6eb2653b079863"
format_stamp: "Formatted at 2026-06-19 13:14:00 on dist-test-slave-mkt3"
server_key: "57c3a1e6741b5e414717f57ef482bfeb"
server_key_iv: "858d4cbcfacf9ea58629805626ba134a"
server_key_version: "encryptionkey@0"
I20260619 13:14:00.252488 4172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ebfaaf81a5af4a129f6eb2653b079863"
format_stamp: "Formatted at 2026-06-19 13:14:00 on dist-test-slave-mkt3"
server_key: "57c3a1e6741b5e414717f57ef482bfeb"
server_key_iv: "858d4cbcfacf9ea58629805626ba134a"
server_key_version: "encryptionkey@0"
I20260619 13:14:00.257241 4172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260619 13:14:00.260442 4187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:00.261710 4172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:14:00.261893 4172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "ebfaaf81a5af4a129f6eb2653b079863"
format_stamp: "Formatted at 2026-06-19 13:14:00 on dist-test-slave-mkt3"
server_key: "57c3a1e6741b5e414717f57ef482bfeb"
server_key_iv: "858d4cbcfacf9ea58629805626ba134a"
server_key_version: "encryptionkey@0"
I20260619 13:14:00.262070 4172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:00.304381 4172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:00.308259 4172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:00.308511 4172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:00.309458 4172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:00.310771 4172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:00.310843 4172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:00.310936 4172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:00.310985 4172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:00.339370 4172 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42749
I20260619 13:14:00.339399 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42749 every 8 connection(s)
I20260619 13:14:00.340767 4172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:14:00.347470 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 4172
I20260619 13:14:00.347666 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:14:00.348173 388 external_mini_cluster.cc:1428] Setting key 7de98bcc5e31746b6d3ddf54dea895c1
Jun 19 13:14:00 dist-test-slave-mkt3 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:00.361342 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:00.343365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32791 (local address 127.0.97.62:46853)
0619 13:14:00.343516 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:00.343521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:00.344805 (+ 1284us) server_negotiation.cc:407] Connection header received
0619 13:14:00.346563 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:00.346569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:00.346649 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:00.346733 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:00.348677 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:00.349474 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:00.351522 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:00.351907 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:00.352004 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:00.355497 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:00.355504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:00.355508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:00.355544 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:00.358518 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:00.359191 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:00.359195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:00.359199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:00.359338 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:00.359726 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:00.359731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:00.359734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:00.360069 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:00.360195 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:00.360823 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 13:14:00.361035 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:14:00.362191 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:00.343901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46853 (local address 127.0.97.2:32791)
0619 13:14:00.344653 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:00.344697 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:00.346307 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:00.346962 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:00.346970 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:00.347733 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:00.348494 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:00.348520 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:00.349620 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:00.349625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:00.351384 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:00.351400 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:00.352106 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:00.352112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:00.352357 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:00.353013 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:00.353038 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:00.355300 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:00.358658 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:00.358664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:00.358676 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:00.359076 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:00.359436 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:00.359439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:00.359442 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:00.359622 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:00.360204 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:00.360209 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:00.360495 (+ 286us) client_negotiation.cc:769] Sending connection context
0619 13:14:00.360708 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 13:14:00.361082 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":309,"threads_started":1}
I20260619 13:14:00.363547 4300 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46853
I20260619 13:14:00.363868 4300 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:00.364702 4300 heartbeater.cc:511] Master 127.0.97.62:46853 requested a full tablet report, sending...
I20260619 13:14:00.366174 3977 ts_manager.cc:194] Registered new tserver with Master: ebfaaf81a5af4a129f6eb2653b079863 (127.0.97.2:42749)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:14:00.366972 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:32791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:14:00.386515 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46853
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:00.686491 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:00.686765 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:00.686883 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:00.693830 4305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:00.693969 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:00.694119 4305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:14:00.703715 4305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:00.705006 4305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:00.706370 4305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:00.718564 4310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:00.718639 4311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:00.718603 4313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:00.719646 4305 server_base.cc:1034] running on GCE node
I20260619 13:14:00.720216 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:00.721096 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:00.722299 4305 hybrid_clock.cc:648] HybridClock initialized: now 1781874840722246 us; error 46 us; skew 500 ppm
I20260619 13:14:00.722582 4305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:00 dist-test-slave-mkt3 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:00.725370 4305 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:14:00.726737 4305 webserver.cc:460] Webserver started at http://127.0.97.3:46233/ using document root <none> and password file <none>
I20260619 13:14:00.727367 4305 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:00.727478 4305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:00.727751 4305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:00.730492 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "7b9146f599864a47805ee5254761a8a9"
format_stamp: "Formatted at 2026-06-19 13:14:00 on dist-test-slave-mkt3"
server_key: "1fa936394a4480ac78221e6eb3c4c3b8"
server_key_iv: "0b9c63a1c8130d9561bf49115d2222f3"
server_key_version: "encryptionkey@0"
I20260619 13:14:00.731202 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7b9146f599864a47805ee5254761a8a9"
format_stamp: "Formatted at 2026-06-19 13:14:00 on dist-test-slave-mkt3"
server_key: "1fa936394a4480ac78221e6eb3c4c3b8"
server_key_iv: "0b9c63a1c8130d9561bf49115d2222f3"
server_key_version: "encryptionkey@0"
I20260619 13:14:00.735906 4305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:14:00.738963 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:00.740166 4305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 13:14:00.740305 4305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "7b9146f599864a47805ee5254761a8a9"
format_stamp: "Formatted at 2026-06-19 13:14:00 on dist-test-slave-mkt3"
server_key: "1fa936394a4480ac78221e6eb3c4c3b8"
server_key_iv: "0b9c63a1c8130d9561bf49115d2222f3"
server_key_version: "encryptionkey@0"
I20260619 13:14:00.740471 4305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:00.762374 4305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:00.766042 4305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:00.766284 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:00.767128 4305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:00.768301 4305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:00.768373 4305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:00.768481 4305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:00.768565 4305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:00.796479 4305 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45461
I20260619 13:14:00.796533 4432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45461 every 8 connection(s)
I20260619 13:14:00.797634 4305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:14:00.801215 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 4305
I20260619 13:14:00.801398 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:14:00.801930 388 external_mini_cluster.cc:1428] Setting key 35831c13606eaa865208344499eee992
Jun 19 13:14:00 dist-test-slave-mkt3 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:00.817229 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:00.799856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36189 (local address 127.0.97.62:46853)
0619 13:14:00.800045 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:00.800051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:00.801316 (+ 1265us) server_negotiation.cc:407] Connection header received
0619 13:14:00.803559 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:00.803564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:00.803660 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:00.803761 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:00.805419 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:00.806213 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:00.807802 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:00.808221 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:00.808324 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:00.811544 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:00.811549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:00.811552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:00.811584 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:00.813987 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:00.814667 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:00.814671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:00.814673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:00.814788 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:00.815288 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:00.815293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:00.815296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:00.815647 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:00.815772 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:00.816730 (+ 958us) server_negotiation.cc:299] Negotiation successful
0619 13:14:00.816929 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:14:00.818168 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:00.800373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46853 (local address 127.0.97.3:36189)
0619 13:14:00.801115 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:00.801165 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:00.803260 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:00.804004 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:00.804012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:00.804594 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:00.805249 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:00.805269 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:00.806382 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:00.806386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:00.807647 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:00.807659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:00.808391 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:00.808395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:00.808582 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:00.809210 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:00.809232 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:00.811331 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:00.814117 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:00.814130 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:00.814146 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:00.814563 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:00.814908 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:00.814912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:00.814915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:00.815149 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:00.815903 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:00.815909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:00.816311 (+ 402us) client_negotiation.cc:769] Sending connection context
0619 13:14:00.816636 (+ 325us) client_negotiation.cc:240] Negotiation successful
0619 13:14:00.817087 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":258,"threads_started":1}
I20260619 13:14:00.819370 4433 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46853
I20260619 13:14:00.819646 4433 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:00.820358 4433 heartbeater.cc:511] Master 127.0.97.62:46853 requested a full tablet report, sending...
I20260619 13:14:00.821741 3977 ts_manager.cc:194] Registered new tserver with Master: 7b9146f599864a47805ee5254761a8a9 (127.0.97.3:45461)
I20260619 13:14:00.822548 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36189
I20260619 13:14:00.829137 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:14:00 dist-test-slave-mkt3 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 13:14:00 dist-test-slave-mkt3 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:14:00.872262 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:00.858070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.0.97.1:40173)
0619 13:14:00.858639 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:00.858648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:00.858678 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:14:00.858737 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:00.858744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:00.858948 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:00.859074 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:00.860317 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:00.861323 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:00.862505 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:00.863059 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:00.863176 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:00.866684 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:00.866695 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:00.866708 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:00.866743 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:00.869575 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:00.870316 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:00.870324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:00.870326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:00.870438 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:00.870806 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:00.870809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:00.870811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:00.871143 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:00.871411 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:00.871701 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 13:14:00.871938 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":330,"threads_started":1}
Jun 19 13:14:00 dist-test-slave-mkt3 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:00.889065 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:00.875957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.0.97.62:46853)
0619 13:14:00.876152 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:00.876158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:00.876323 (+ 165us) server_negotiation.cc:407] Connection header received
0619 13:14:00.876470 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:00.876475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:00.876540 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:00.876614 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:00.877756 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:00.878549 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:00.879718 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:00.880127 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:00.880232 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:00.884316 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:00.884321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:00.884324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:00.884355 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:00.886756 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:00.887312 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:00.887317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:00.887321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:00.887435 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:00.887751 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:00.887756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:00.887760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:00.888055 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:00.888265 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:00.888569 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 13:14:00.888777 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 13:14:00.889605 3977 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51058
I20260619 13:14:00.890525 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 4038
I20260619 13:14:00.898584 4166 heartbeater.cc:503] Master 127.0.97.62:46853 was elected leader, sending a full tablet report...
I20260619 13:14:00.902626 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:00.891432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.0.97.1:40173)
0619 13:14:00.891622 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:00.891628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:00.891750 (+ 122us) server_negotiation.cc:407] Connection header received
0619 13:14:00.891921 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:00.891927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:00.892011 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:00.892089 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:00.893240 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:00.893974 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:00.895099 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:00.895521 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:00.895631 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:00.897638 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:00.897645 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:00.897648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:00.897678 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:00.900388 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:00.900926 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:00.900932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:00.900936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:00.901054 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:00.901379 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:00.901384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:00.901387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:00.901620 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:00.901829 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:00.902140 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 13:14:00.902351 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:14:00.903153 4161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:00.987967 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 4038
I20260619 13:14:01.001098 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 4172
Jun 19 13:14:01 dist-test-slave-mkt3 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:14:01.016170 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:01.002004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.0.97.2:42749)
0619 13:14:01.002473 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:01.002480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:01.002504 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:14:01.002560 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:01.002566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:01.002794 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:01.002934 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:01.004133 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:01.005125 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:01.006468 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:01.007012 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:01.007132 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:01.010967 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:01.010977 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:01.010992 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:01.011025 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:01.013623 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:01.014382 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:01.014390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:01.014392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:01.014516 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:01.014830 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:01.014833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:01.014836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:01.015174 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:01.015387 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:01.015679 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 13:14:01.015872 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":224,"threads_started":1}
I20260619 13:14:01.017438 4295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:01.100675 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 4172
I20260619 13:14:01.112663 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 4305
Jun 19 13:14:01 dist-test-slave-mkt3 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:14:01.127308 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:01.113490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39134 (local address 127.0.97.3:45461)
0619 13:14:01.114091 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:01.114099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:01.114125 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:14:01.114183 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:01.114189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:01.114400 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:01.114545 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:01.115731 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:01.116721 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:01.117990 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:01.118525 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:01.118629 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:01.122474 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:01.122484 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:01.122499 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:01.122533 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:01.125077 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:01.125598 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:01.125615 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:01.125617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:01.125726 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:01.126028 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:01.126031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:01.126033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:01.126362 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:01.126582 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:01.126789 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 13:14:01.126987 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":303,"threads_started":1}
I20260619 13:14:01.128477 4428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:01.206643 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 4305
I20260619 13:14:01.218673 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 3945
I20260619 13:14:01.219732 4007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:01.317762 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 3945
2026-06-19T13:14:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3569 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:14:01 dist-test-slave-mkt3 krb5kdc[4452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:14:01 dist-test-slave-mkt3 krb5kdc[4452](info): set up 2 sockets
Jun 19 13:14:01 dist-test-slave-mkt3 krb5kdc[4452](info): commencing operation
krb5kdc: starting...
W20260619 13:14:02.309768 388 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:14:02 dist-test-slave-mkt3 krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:14:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:14:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:14:02.476855 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37571
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:02.775597 4465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:02.775877 4465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:02.775995 4465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:02.783198 4465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:02.783309 4465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:02.783355 4465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:02.783432 4465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:02.783490 4465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:14:02.793094 4465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:02.794310 4465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:02.795578 4465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:02.806092 4471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:02.806120 4470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:02.806314 4473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:02.806874 4465 server_base.cc:1034] running on GCE node
I20260619 13:14:02.807574 4465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:02.808820 4465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:02.810011 4465 hybrid_clock.cc:648] HybridClock initialized: now 1781874842809967 us; error 37 us; skew 500 ppm
I20260619 13:14:02.810338 4465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:02 dist-test-slave-mkt3 krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:02.813086 4465 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:14:02.814221 4465 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:02.814304 4465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:02.814550 4465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:02.817251 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "b306b130a91347ffb40e17db0df0669d"
format_stamp: "Formatted at 2026-06-19 13:14:02 on dist-test-slave-mkt3"
server_key: "8d40b49bcc2249b6c3e8a14e65da408d"
server_key_iv: "f90d4493edda4bee429ddb8f3a2d0c86"
server_key_version: "encryptionkey@0"
I20260619 13:14:02.817993 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "b306b130a91347ffb40e17db0df0669d"
format_stamp: "Formatted at 2026-06-19 13:14:02 on dist-test-slave-mkt3"
server_key: "8d40b49bcc2249b6c3e8a14e65da408d"
server_key_iv: "f90d4493edda4bee429ddb8f3a2d0c86"
server_key_version: "encryptionkey@0"
I20260619 13:14:02.822577 4465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:14:02.825641 4479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:02.826855 4465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 13:14:02.826979 4465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "b306b130a91347ffb40e17db0df0669d"
format_stamp: "Formatted at 2026-06-19 13:14:02 on dist-test-slave-mkt3"
server_key: "8d40b49bcc2249b6c3e8a14e65da408d"
server_key_iv: "f90d4493edda4bee429ddb8f3a2d0c86"
server_key_version: "encryptionkey@0"
I20260619 13:14:02.827147 4465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:02.838028 4465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:02.841476 4465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:02.841683 4465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:02.857316 4465 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37571
I20260619 13:14:02.857336 4530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37571 every 8 connection(s)
I20260619 13:14:02.858103 4465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:14:02.861457 4531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:02.862653 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 4465
I20260619 13:14:02.862843 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:14:02.863420 388 external_mini_cluster.cc:1428] Setting key a76a9eb1e608639ce9c28b644ff06aa7
I20260619 13:14:02.874116 4531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d: Bootstrap starting.
Jun 19 13:14:02 dist-test-slave-mkt3 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:02.877935 4531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:02.879017 4531 log.cc:824] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:02.881796 4531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d: No bootstrap required, opened a new log
I20260619 13:14:02.884963 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:02.865128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59448 (local address 127.0.97.62:37571)
0619 13:14:02.865941 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:02.865952 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:02.865990 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:14:02.867583 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:02.867604 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:02.867958 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:02.868239 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:02.869988 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:02.871287 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:02.873011 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:02.873634 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:02.873807 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:02.878354 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:02.878361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:02.878378 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:02.878423 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:02.881146 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:02.881932 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:02.881941 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:02.881948 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:02.882103 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:02.882469 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:02.882472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:02.882474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:02.882947 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:02.883155 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:02.883606 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 13:14:02.883958 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":343,"threads_started":1}
I20260619 13:14:02.890090 4531 raft_consensus.cc:348] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b306b130a91347ffb40e17db0df0669d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37571 } }
I20260619 13:14:02.890522 4531 raft_consensus.cc:374] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:02.890630 4531 raft_consensus.cc:729] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b306b130a91347ffb40e17db0df0669d, State: Initialized, Role: FOLLOWER
I20260619 13:14:02.891340 4531 consensus_queue.cc:260] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b306b130a91347ffb40e17db0df0669d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37571 } }
I20260619 13:14:02.891541 4531 raft_consensus.cc:388] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:14:02.891664 4531 raft_consensus.cc:482] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:14:02.891795 4531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:02.894089 4531 raft_consensus.cc:504] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b306b130a91347ffb40e17db0df0669d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37571 } }
I20260619 13:14:02.894570 4531 leader_election.cc:302] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b306b130a91347ffb40e17db0df0669d; no voters:
I20260619 13:14:02.895067 4531 leader_election.cc:288] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:14:02.895174 4536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:02.895748 4536 raft_consensus.cc:686] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [term 1 LEADER]: Becoming Leader. State: Replica: b306b130a91347ffb40e17db0df0669d, State: Running, Role: LEADER
I20260619 13:14:02.896262 4536 consensus_queue.cc:237] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b306b130a91347ffb40e17db0df0669d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37571 } }
I20260619 13:14:02.896942 4531 sys_catalog.cc:564] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:02.898731 4538 sys_catalog.cc:455] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b306b130a91347ffb40e17db0df0669d. Latest consensus state: current_term: 1 leader_uuid: "b306b130a91347ffb40e17db0df0669d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b306b130a91347ffb40e17db0df0669d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37571 } } }
I20260619 13:14:02.899081 4538 sys_catalog.cc:458] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:02.899566 4537 sys_catalog.cc:455] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b306b130a91347ffb40e17db0df0669d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b306b130a91347ffb40e17db0df0669d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37571 } } }
I20260619 13:14:02.900334 4537 sys_catalog.cc:458] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:02.901049 4541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:14:02.905926 4541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:14:02.914745 4541 catalog_manager.cc:1347] Generated new cluster ID: 8e3e55f6319749f39d30886a1783a903
I20260619 13:14:02.914842 4541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:14:02.945425 4541 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:14:02.946782 4541 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:14:02.956393 4541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d: Generated new TSK 0
I20260619 13:14:02.957191 4541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:14:03.003149 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37571
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:03.342316 4557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:03.342585 4557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:03.342739 4557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:03.349725 4557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:03.349831 4557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:03.349996 4557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:14:03.359468 4557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:03.360662 4557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:03.361922 4557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:03.374809 4563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:03.374866 4562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:03.374866 4565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:03.375978 4557 server_base.cc:1034] running on GCE node
I20260619 13:14:03.376535 4557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:03.377384 4557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:03.378580 4557 hybrid_clock.cc:648] HybridClock initialized: now 1781874843378534 us; error 38 us; skew 500 ppm
I20260619 13:14:03.378865 4557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:03 dist-test-slave-mkt3 krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:03.381824 4557 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:14:03.382896 4557 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:03.382983 4557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:03.383271 4557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:03.386313 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "2b709df8d36d44d79b35e3aa481b8c14"
format_stamp: "Formatted at 2026-06-19 13:14:03 on dist-test-slave-mkt3"
server_key: "bd9409f348dece8ae71e0f159b129f37"
server_key_iv: "5ff6ece0e6bf5931b5f4930c3eb02a2e"
server_key_version: "encryptionkey@0"
I20260619 13:14:03.387072 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2b709df8d36d44d79b35e3aa481b8c14"
format_stamp: "Formatted at 2026-06-19 13:14:03 on dist-test-slave-mkt3"
server_key: "bd9409f348dece8ae71e0f159b129f37"
server_key_iv: "5ff6ece0e6bf5931b5f4930c3eb02a2e"
server_key_version: "encryptionkey@0"
I20260619 13:14:03.391739 4557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 13:14:03.395067 4571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:03.396310 4557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:14:03.396481 4557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "2b709df8d36d44d79b35e3aa481b8c14"
format_stamp: "Formatted at 2026-06-19 13:14:03 on dist-test-slave-mkt3"
server_key: "bd9409f348dece8ae71e0f159b129f37"
server_key_iv: "5ff6ece0e6bf5931b5f4930c3eb02a2e"
server_key_version: "encryptionkey@0"
I20260619 13:14:03.396692 4557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:03.424808 4557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:03.428565 4557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:03.428807 4557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:03.429755 4557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:03.431061 4557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:03.431151 4557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:03.431249 4557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:03.431316 4557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:03.459266 4557 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32923
I20260619 13:14:03.459306 4683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32923 every 8 connection(s)
I20260619 13:14:03.460021 4557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:14:03.462566 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 4557
I20260619 13:14:03.462762 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:14:03.463564 388 external_mini_cluster.cc:1428] Setting key 97be23d962f4e4a0cd34253fb138b51d
Jun 19 13:14:03 dist-test-slave-mkt3 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:14:03.478734 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:03.462264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34993 (local address 127.0.97.62:37571)
0619 13:14:03.462646 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:03.462652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:03.463778 (+ 1126us) server_negotiation.cc:407] Connection header received
0619 13:14:03.465908 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:03.465913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:03.465987 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:03.466088 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:03.467470 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:03.468330 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:03.470321 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:03.470689 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:03.470796 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:03.473778 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:03.473783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:03.473786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:03.473815 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:03.476080 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:03.476692 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:03.476698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:03.476702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:03.476809 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:03.477161 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:03.477166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:03.477170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:03.477419 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:03.477539 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:03.478191 (+ 652us) server_negotiation.cc:299] Negotiation successful
0619 13:14:03.478401 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":192,"threads_started":1}
I20260619 13:14:03.479612 4687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:03.462918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37571 (local address 127.0.97.1:34993)
0619 13:14:03.463613 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:03.463667 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:03.465656 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:03.466308 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:03.466316 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:03.466792 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:03.467321 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:03.467337 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:03.468483 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:03.468489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:03.470184 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:03.470196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:03.470888 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:03.470895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:03.471104 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:03.471669 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:03.471690 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:03.473590 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:03.476198 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:03.476208 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:03.476223 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:03.476589 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:03.476900 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:03.476905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:03.476909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:03.477066 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:03.477528 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:03.477536 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:03.477832 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 13:14:03.478103 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 13:14:03.478549 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":279,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:14:03.481092 4684 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37571
I20260619 13:14:03.481426 4684 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:03.482223 4684 heartbeater.cc:511] Master 127.0.97.62:37571 requested a full tablet report, sending...
I20260619 13:14:03.484323 4496 ts_manager.cc:194] Registered new tserver with Master: 2b709df8d36d44d79b35e3aa481b8c14 (127.0.97.1:32923)
I20260619 13:14:03.486043 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:14:03.501147 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37571
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:03.798797 4690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:03.799052 4690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:03.799178 4690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:03.806404 4690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:03.806500 4690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:03.806638 4690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:14:03.815856 4690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:03.817103 4690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:03.818364 4690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:03.830094 4696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:03.830158 4695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:03.830159 4698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:03.830906 4690 server_base.cc:1034] running on GCE node
I20260619 13:14:03.831437 4690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:03.832262 4690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:03.833444 4690 hybrid_clock.cc:648] HybridClock initialized: now 1781874843833395 us; error 48 us; skew 500 ppm
I20260619 13:14:03.833719 4690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:03 dist-test-slave-mkt3 krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:03.836539 4690 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:14:03.837522 4690 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:03.837613 4690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:03.837944 4690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:03.840592 4690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "4887ccc5e56f49fbb5e51899feb0b78a"
format_stamp: "Formatted at 2026-06-19 13:14:03 on dist-test-slave-mkt3"
server_key: "fbf3bb595b52633c518cfea5faa0fa3a"
server_key_iv: "02e8af72f11ca2d4b5b5494e918f24de"
server_key_version: "encryptionkey@0"
I20260619 13:14:03.841351 4690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4887ccc5e56f49fbb5e51899feb0b78a"
format_stamp: "Formatted at 2026-06-19 13:14:03 on dist-test-slave-mkt3"
server_key: "fbf3bb595b52633c518cfea5faa0fa3a"
server_key_iv: "02e8af72f11ca2d4b5b5494e918f24de"
server_key_version: "encryptionkey@0"
I20260619 13:14:03.846009 4690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260619 13:14:03.849149 4704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:03.850402 4690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:14:03.850538 4690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "4887ccc5e56f49fbb5e51899feb0b78a"
format_stamp: "Formatted at 2026-06-19 13:14:03 on dist-test-slave-mkt3"
server_key: "fbf3bb595b52633c518cfea5faa0fa3a"
server_key_iv: "02e8af72f11ca2d4b5b5494e918f24de"
server_key_version: "encryptionkey@0"
I20260619 13:14:03.850711 4690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:03.883126 4690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:03.886987 4690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:03.887226 4690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:03.888092 4690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:03.889295 4690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:03.889384 4690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:03.889483 4690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:03.889549 4690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:03.919186 4690 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43649
I20260619 13:14:03.919227 4816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43649 every 8 connection(s)
I20260619 13:14:03.919936 4690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:14:03.925974 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 4690
I20260619 13:14:03.926143 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:14:03.926558 388 external_mini_cluster.cc:1428] Setting key d1d99173717849167ba6d48fd08ad010
Jun 19 13:14:03 dist-test-slave-mkt3 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:03.938753 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:03.922199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34853 (local address 127.0.97.62:37571)
0619 13:14:03.922364 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:03.922369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:03.923482 (+ 1113us) server_negotiation.cc:407] Connection header received
0619 13:14:03.925132 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:03.925136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:03.925202 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:03.925286 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:03.926774 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:03.927633 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:03.929594 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:03.930051 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:03.930155 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:03.933708 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:03.933715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:03.933719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:03.933757 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:03.936043 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:03.936789 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:03.936792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:03.936794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:03.936899 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:03.937288 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:03.937291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:03.937293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:03.937519 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:03.937613 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:03.938370 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 13:14:03.938526 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:14:03.940084 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:03.922692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37571 (local address 127.0.97.2:34853)
0619 13:14:03.923334 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:03.923371 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:03.924904 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:03.925457 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:03.925465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:03.925993 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:03.926608 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:03.926629 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:03.927767 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:03.927771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:03.929462 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:03.929476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:03.930269 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:03.930276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:03.930531 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:03.931207 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:03.931233 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:03.933501 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:03.936166 (+ 2665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:03.936174 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:03.936189 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:03.936700 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:03.936983 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:03.936988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:03.936991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:03.937199 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:03.937622 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:03.937629 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:03.938017 (+ 388us) client_negotiation.cc:769] Sending connection context
0619 13:14:03.938311 (+ 294us) client_negotiation.cc:240] Negotiation successful
0619 13:14:03.938776 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":241,"threads_started":1}
I20260619 13:14:03.941473 4817 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37571
I20260619 13:14:03.941752 4817 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:03.942396 4817 heartbeater.cc:511] Master 127.0.97.62:37571 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:14:03.943821 4496 ts_manager.cc:194] Registered new tserver with Master: 4887ccc5e56f49fbb5e51899feb0b78a (127.0.97.2:43649)
I20260619 13:14:03.944628 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:14:03.964622 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37571
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:04.267160 4822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:04.267443 4822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:04.267580 4822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:04.274544 4822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:04.274646 4822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:04.274785 4822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:14:04.283982 4822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:04.285183 4822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:04.286422 4822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:04.298148 4827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:04.298115 4828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:04.298174 4830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:04.298769 4822 server_base.cc:1034] running on GCE node
I20260619 13:14:04.299316 4822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:04.300247 4822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:04.301435 4822 hybrid_clock.cc:648] HybridClock initialized: now 1781874844301371 us; error 59 us; skew 500 ppm
I20260619 13:14:04.301734 4822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:04 dist-test-slave-mkt3 krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:04.308295 4822 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:14:04.309366 4822 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:04.309451 4822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:04.309712 4822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:04.312479 4822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "d196eb606f204717978c729f9be43863"
format_stamp: "Formatted at 2026-06-19 13:14:04 on dist-test-slave-mkt3"
server_key: "161a6404e7e9098bd8cacec24d18a4c8"
server_key_iv: "e871af7afe9b3ab19237fa60f3dbefa0"
server_key_version: "encryptionkey@0"
I20260619 13:14:04.313199 4822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d196eb606f204717978c729f9be43863"
format_stamp: "Formatted at 2026-06-19 13:14:04 on dist-test-slave-mkt3"
server_key: "161a6404e7e9098bd8cacec24d18a4c8"
server_key_iv: "e871af7afe9b3ab19237fa60f3dbefa0"
server_key_version: "encryptionkey@0"
I20260619 13:14:04.317804 4822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260619 13:14:04.321025 4836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:04.322321 4822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:14:04.322454 4822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "d196eb606f204717978c729f9be43863"
format_stamp: "Formatted at 2026-06-19 13:14:04 on dist-test-slave-mkt3"
server_key: "161a6404e7e9098bd8cacec24d18a4c8"
server_key_iv: "e871af7afe9b3ab19237fa60f3dbefa0"
server_key_version: "encryptionkey@0"
I20260619 13:14:04.322631 4822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:04.373281 4822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:04.376917 4822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:04.377147 4822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:04.378039 4822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:04.379273 4822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:04.379359 4822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:04.379456 4822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:04.379521 4822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:04.407895 4822 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43157
I20260619 13:14:04.407948 4948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43157 every 8 connection(s)
I20260619 13:14:04.408671 4822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:14:04.409658 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 4822
I20260619 13:14:04.409860 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:14:04.410372 388 external_mini_cluster.cc:1428] Setting key 3c304e2ecdc323a1f2e0e4e867328ee2
Jun 19 13:14:04 dist-test-slave-mkt3 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:04.427014 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.411097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44715 (local address 127.0.97.62:37571)
0619 13:14:04.411304 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:04.411310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:04.412573 (+ 1263us) server_negotiation.cc:407] Connection header received
0619 13:14:04.414443 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:04.414448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:04.414525 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:04.414609 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:04.415974 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.416794 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.418272 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.418637 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.418749 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.421802 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:04.421807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:04.421810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:04.421865 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:04.424301 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:04.424948 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:04.424954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:04.424956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:04.425063 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:04.425474 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:04.425478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:04.425480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:04.425761 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:04.425913 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:04.426537 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 13:14:04.426739 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:14:04.427836 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.411719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37571 (local address 127.0.97.3:44715)
0619 13:14:04.412441 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:04.412480 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:04.414211 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:04.414821 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:04.414829 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:04.415299 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:04.415821 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:04.415837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.416919 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.416923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:04.418143 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:04.418154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.418829 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.418835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:04.419046 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.419627 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:04.419648 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:04.421611 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:04.424424 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:04.424432 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:04.424448 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:04.424843 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:04.425159 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:04.425164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:04.425168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:04.425369 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:04.425894 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:04.425903 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:04.426208 (+ 305us) client_negotiation.cc:769] Sending connection context
0619 13:14:04.426445 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 13:14:04.426819 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":283,"threads_started":1}
I20260619 13:14:04.429132 4949 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37571
I20260619 13:14:04.429394 4949 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:04.430014 4949 heartbeater.cc:511] Master 127.0.97.62:37571 requested a full tablet report, sending...
I20260619 13:14:04.431358 4496 ts_manager.cc:194] Registered new tserver with Master: d196eb606f204717978c729f9be43863 (127.0.97.3:43157)
I20260619 13:14:04.432121 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44715
I20260619 13:14:04.437326 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:14:04.452646 4686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.441519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.0.97.62:37571)
0619 13:14:04.441677 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:04.441682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:04.442135 (+ 453us) server_negotiation.cc:407] Connection header received
0619 13:14:04.442315 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:04.442320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:04.442394 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:04.442471 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:04.443752 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.444420 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.445602 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.446085 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.446192 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.448163 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:04.448169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:04.448173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:04.448202 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:04.450340 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:04.450893 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:04.450899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:04.450902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:04.451009 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:04.451338 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:04.451343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:04.451346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:04.451628 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:04.451871 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:04.452150 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 13:14:04.452386 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:14:04.456102 4496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:14:04.462566 4496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:14:04.481980 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.473035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.0.97.1:32923)
0619 13:14:04.474401 (+ 1366us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:04.474409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:04.474437 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:14:04.474525 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:04.474532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:04.474775 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:04.474949 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:04.475745 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.477240 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.479583 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.480910 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.481038 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.481257 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:04.481511 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 13:14:04.481668 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1214,"thread_start_us":1091,"threads_started":1}
I20260619 13:14:04.482084 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.473016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32923 (local address 127.0.0.1:36512)
0619 13:14:04.473714 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:04.473754 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:04.473934 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:04.475206 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:04.475213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:04.475242 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:04.475578 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:04.475589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.477359 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.477364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:04.479387 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:04.479398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.481095 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.481101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:04.481296 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.481345 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 13:14:04.481461 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 13:14:04.481624 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":1,"spinlock_wait_cycles":6528,"thread_start_us":346,"threads_started":1}
I20260619 13:14:04.482750 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.474117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43157 (local address 127.0.0.1:46338)
0619 13:14:04.474632 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:04.474647 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:04.474744 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:04.476343 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:04.476348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:04.476371 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:04.476804 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:04.476816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.478379 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.478383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:04.480339 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:04.480352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.481984 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.481991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:04.482159 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.482187 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 13:14:04.482246 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 13:14:04.482360 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":32,"thread_start_us":291,"threads_started":1}
I20260619 13:14:04.483101 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.474708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43649 (local address 127.0.0.1:52330)
0619 13:14:04.475161 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:04.475176 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:04.475276 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:04.476601 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:04.476605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:04.476632 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:04.477023 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:04.477033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.478958 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.478963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:04.480787 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:04.480797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.482423 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.482429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:04.482585 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.482619 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 13:14:04.482697 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 13:14:04.482812 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":659,"threads_started":1}
I20260619 13:14:04.483374 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.474522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.0.97.3:43157)
0619 13:14:04.475785 (+ 1263us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:04.475793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:04.475823 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:14:04.475907 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:04.475914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:04.476135 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:04.477000 (+ 865us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:04.477246 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.478262 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.480486 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.481742 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.482632 (+ 890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.482854 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:04.482916 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 13:14:04.483058 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1110,"thread_start_us":952,"threads_started":1}
I20260619 13:14:04.483952 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.475199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.0.97.2:43649)
0619 13:14:04.476040 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:04.476047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:04.476073 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:14:04.476181 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:04.476188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:04.476409 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:04.477164 (+ 755us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:04.477320 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.478821 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.480911 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.482252 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.482366 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.482533 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:04.483340 (+ 807us) server_negotiation.cc:299] Negotiation successful
0619 13:14:04.483661 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":587,"threads_started":1}
I20260619 13:14:04.485918 4619 tablet_service.cc:1467] Processing CreateTablet for tablet d43c7b00a5494523908638338e12778a (DEFAULT_TABLE table=test-table [id=f535315e1a7d4b2ea65e476bdf09e866]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:04.486987 4619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d43c7b00a5494523908638338e12778a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:04.486824 4884 tablet_service.cc:1467] Processing CreateTablet for tablet d43c7b00a5494523908638338e12778a (DEFAULT_TABLE table=test-table [id=f535315e1a7d4b2ea65e476bdf09e866]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:04.487200 4752 tablet_service.cc:1467] Processing CreateTablet for tablet d43c7b00a5494523908638338e12778a (DEFAULT_TABLE table=test-table [id=f535315e1a7d4b2ea65e476bdf09e866]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:04.488102 4884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d43c7b00a5494523908638338e12778a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:04.488371 4752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d43c7b00a5494523908638338e12778a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:04.488698 4684 heartbeater.cc:503] Master 127.0.97.62:37571 was elected leader, sending a full tablet report...
I20260619 13:14:04.498188 4967 tablet_bootstrap.cc:492] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14: Bootstrap starting.
I20260619 13:14:04.501456 4967 tablet_bootstrap.cc:654] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:04.502570 4967 log.cc:824] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:04.503862 4968 tablet_bootstrap.cc:492] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a: Bootstrap starting.
I20260619 13:14:04.504580 4969 tablet_bootstrap.cc:492] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863: Bootstrap starting.
I20260619 13:14:04.505707 4967 tablet_bootstrap.cc:492] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14: No bootstrap required, opened a new log
I20260619 13:14:04.506013 4967 ts_tablet_manager.cc:1397] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260619 13:14:04.508037 4968 tablet_bootstrap.cc:654] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:04.508304 4969 tablet_bootstrap.cc:654] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:04.509217 4969 log.cc:824] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:04.509227 4968 log.cc:824] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:04.511952 4969 tablet_bootstrap.cc:492] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863: No bootstrap required, opened a new log
I20260619 13:14:04.512210 4969 ts_tablet_manager.cc:1397] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:14:04.513828 4968 tablet_bootstrap.cc:492] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a: No bootstrap required, opened a new log
I20260619 13:14:04.514091 4968 ts_tablet_manager.cc:1397] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 13:14:04.515744 4967 raft_consensus.cc:348] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b709df8d36d44d79b35e3aa481b8c14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32923 } } peers { permanent_uuid: "d196eb606f204717978c729f9be43863" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43157 } } peers { permanent_uuid: "4887ccc5e56f49fbb5e51899feb0b78a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43649 } }
I20260619 13:14:04.516155 4967 raft_consensus.cc:374] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:04.516290 4967 raft_consensus.cc:729] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b709df8d36d44d79b35e3aa481b8c14, State: Initialized, Role: FOLLOWER
I20260619 13:14:04.516872 4967 consensus_queue.cc:260] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b709df8d36d44d79b35e3aa481b8c14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32923 } } peers { permanent_uuid: "d196eb606f204717978c729f9be43863" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43157 } } peers { permanent_uuid: "4887ccc5e56f49fbb5e51899feb0b78a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43649 } }
I20260619 13:14:04.518533 4967 ts_tablet_manager.cc:1428] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260619 13:14:04.519744 4969 raft_consensus.cc:348] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b709df8d36d44d79b35e3aa481b8c14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32923 } } peers { permanent_uuid: "d196eb606f204717978c729f9be43863" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43157 } } peers { permanent_uuid: "4887ccc5e56f49fbb5e51899feb0b78a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43649 } }
I20260619 13:14:04.520032 4969 raft_consensus.cc:374] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:04.520128 4969 raft_consensus.cc:729] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d196eb606f204717978c729f9be43863, State: Initialized, Role: FOLLOWER
I20260619 13:14:04.520659 4969 consensus_queue.cc:260] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b709df8d36d44d79b35e3aa481b8c14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32923 } } peers { permanent_uuid: "d196eb606f204717978c729f9be43863" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43157 } } peers { permanent_uuid: "4887ccc5e56f49fbb5e51899feb0b78a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43649 } }
I20260619 13:14:04.521616 4949 heartbeater.cc:503] Master 127.0.97.62:37571 was elected leader, sending a full tablet report...
I20260619 13:14:04.522239 4969 ts_tablet_manager.cc:1428] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260619 13:14:04.523852 4968 raft_consensus.cc:348] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b709df8d36d44d79b35e3aa481b8c14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32923 } } peers { permanent_uuid: "d196eb606f204717978c729f9be43863" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43157 } } peers { permanent_uuid: "4887ccc5e56f49fbb5e51899feb0b78a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43649 } }
I20260619 13:14:04.524250 4968 raft_consensus.cc:374] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:04.524346 4968 raft_consensus.cc:729] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4887ccc5e56f49fbb5e51899feb0b78a, State: Initialized, Role: FOLLOWER
I20260619 13:14:04.524883 4968 consensus_queue.cc:260] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b709df8d36d44d79b35e3aa481b8c14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32923 } } peers { permanent_uuid: "d196eb606f204717978c729f9be43863" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43157 } } peers { permanent_uuid: "4887ccc5e56f49fbb5e51899feb0b78a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43649 } }
I20260619 13:14:04.525872 4817 heartbeater.cc:503] Master 127.0.97.62:37571 was elected leader, sending a full tablet report...
I20260619 13:14:04.526331 4968 ts_tablet_manager.cc:1428] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260619 13:14:04.607131 4974 raft_consensus.cc:482] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:14:04.607414 4974 raft_consensus.cc:504] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b709df8d36d44d79b35e3aa481b8c14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32923 } } peers { permanent_uuid: "d196eb606f204717978c729f9be43863" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43157 } } peers { permanent_uuid: "4887ccc5e56f49fbb5e51899feb0b78a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43649 } }
I20260619 13:14:04.608955 4974 leader_election.cc:288] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b709df8d36d44d79b35e3aa481b8c14 (127.0.97.1:32923), 4887ccc5e56f49fbb5e51899feb0b78a (127.0.97.2:43649)
I20260619 13:14:04.614553 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.609312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32923 (local address 127.0.97.3:41515)
0619 13:14:04.609521 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:04.609537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:04.609633 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:04.609995 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:04.609999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:04.610026 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:04.610229 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:04.610242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.611348 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.611352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:04.612979 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:04.612996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.614047 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.614051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:04.614179 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.614201 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:14:04.614271 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 13:14:04.614355 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":36}
I20260619 13:14:04.614651 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.609406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41515 (local address 127.0.97.1:32923)
0619 13:14:04.609612 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:04.609618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:04.609638 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:14:04.609762 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:04.609768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:04.609864 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:04.609958 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:04.610379 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.611229 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.613147 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.613917 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.614047 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.614153 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:04.614306 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 13:14:04.614422 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 13:14:04.615666 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.609894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45167 (local address 127.0.97.2:43649)
0619 13:14:04.610076 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:04.610081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:04.610624 (+ 543us) server_negotiation.cc:407] Connection header received
0619 13:14:04.610700 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:04.610705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:04.610776 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:04.610862 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:04.611253 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.611945 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.613876 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.614935 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.615064 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.615144 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:04.615400 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 13:14:04.615490 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:14:04.615833 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.609721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43649 (local address 127.0.97.3:45167)
0619 13:14:04.610515 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:04.610527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:04.610616 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:04.610865 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:04.610868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:04.610882 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:04.611112 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:04.611120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.612081 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.612086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:04.613704 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:04.613715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.615100 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.615104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:04.615265 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.615296 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:14:04.615387 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 13:14:04.615499 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":3,"thread_start_us":500,"threads_started":1}
I20260619 13:14:04.615619 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43c7b00a5494523908638338e12778a" candidate_uuid: "d196eb606f204717978c729f9be43863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b709df8d36d44d79b35e3aa481b8c14" is_pre_election: true
I20260619 13:14:04.616171 4639 raft_consensus.cc:2454] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d196eb606f204717978c729f9be43863 in term 0.
I20260619 13:14:04.616829 4838 leader_election.cc:302] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b709df8d36d44d79b35e3aa481b8c14, d196eb606f204717978c729f9be43863; no voters:
I20260619 13:14:04.616734 4772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43c7b00a5494523908638338e12778a" candidate_uuid: "d196eb606f204717978c729f9be43863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4887ccc5e56f49fbb5e51899feb0b78a" is_pre_election: true
I20260619 13:14:04.617192 4974 raft_consensus.cc:2781] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:14:04.617271 4772 raft_consensus.cc:2454] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d196eb606f204717978c729f9be43863 in term 0.
I20260619 13:14:04.617353 4974 raft_consensus.cc:482] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:14:04.617439 4974 raft_consensus.cc:3037] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:04.619616 4974 raft_consensus.cc:504] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b709df8d36d44d79b35e3aa481b8c14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32923 } } peers { permanent_uuid: "d196eb606f204717978c729f9be43863" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43157 } } peers { permanent_uuid: "4887ccc5e56f49fbb5e51899feb0b78a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43649 } }
I20260619 13:14:04.620151 4974 leader_election.cc:288] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [CANDIDATE]: Term 1 election: Requested vote from peers 2b709df8d36d44d79b35e3aa481b8c14 (127.0.97.1:32923), 4887ccc5e56f49fbb5e51899feb0b78a (127.0.97.2:43649)
I20260619 13:14:04.620564 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43c7b00a5494523908638338e12778a" candidate_uuid: "d196eb606f204717978c729f9be43863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b709df8d36d44d79b35e3aa481b8c14"
I20260619 13:14:04.620638 4772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d43c7b00a5494523908638338e12778a" candidate_uuid: "d196eb606f204717978c729f9be43863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4887ccc5e56f49fbb5e51899feb0b78a"
I20260619 13:14:04.620718 4639 raft_consensus.cc:3037] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:04.620821 4772 raft_consensus.cc:3037] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:04.622908 4639 raft_consensus.cc:2454] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d196eb606f204717978c729f9be43863 in term 1.
I20260619 13:14:04.623232 4772 raft_consensus.cc:2454] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d196eb606f204717978c729f9be43863 in term 1.
I20260619 13:14:04.623232 4838 leader_election.cc:302] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b709df8d36d44d79b35e3aa481b8c14, d196eb606f204717978c729f9be43863; no voters:
I20260619 13:14:04.623523 4974 raft_consensus.cc:2781] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:04.623929 4974 raft_consensus.cc:686] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [term 1 LEADER]: Becoming Leader. State: Replica: d196eb606f204717978c729f9be43863, State: Running, Role: LEADER
I20260619 13:14:04.624383 4974 consensus_queue.cc:237] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b709df8d36d44d79b35e3aa481b8c14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32923 } } peers { permanent_uuid: "d196eb606f204717978c729f9be43863" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43157 } } peers { permanent_uuid: "4887ccc5e56f49fbb5e51899feb0b78a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43649 } }
I20260619 13:14:04.628530 4496 catalog_manager.cc:5495] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 reported cstate change: term changed from 0 to 1, leader changed from <none> to d196eb606f204717978c729f9be43863 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d196eb606f204717978c729f9be43863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b709df8d36d44d79b35e3aa481b8c14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d196eb606f204717978c729f9be43863" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4887ccc5e56f49fbb5e51899feb0b78a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43649 } health_report { overall_health: UNKNOWN } } }
W20260619 13:14:04.660032 4950 tablet.cc:2369] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:14:04.671378 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.666589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.0.97.3:43157)
0619 13:14:04.666785 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:04.666791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:04.666916 (+ 125us) server_negotiation.cc:407] Connection header received
0619 13:14:04.667133 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:04.667137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:04.667228 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:04.667349 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:14:04.667643 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.668345 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.669529 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.669952 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.670041 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.670451 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:14:04.670921 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:14:04.671020 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:04.671078 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 13:14:04.671142 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 13:14:04.671564 4818 tablet.cc:2369] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:14:04.678753 4772 raft_consensus.cc:1261] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a [term 1 FOLLOWER]: Refusing update from remote peer d196eb606f204717978c729f9be43863: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:04.678810 4639 raft_consensus.cc:1261] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14 [term 1 FOLLOWER]: Refusing update from remote peer d196eb606f204717978c729f9be43863: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:04.679593 4977 consensus_queue.cc:1035] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4887ccc5e56f49fbb5e51899feb0b78a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:04.679940 4974 consensus_queue.cc:1035] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b709df8d36d44d79b35e3aa481b8c14" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:04.696703 4981 mvcc.cc:205] Tried to move back new op lower bound from 7298559363794935808 to 7298559363584909312. Current Snapshot: MvccSnapshot[applied={T|T < 7298559363794935808 or (T in {7298559363794935808})}]
I20260619 13:14:04.704836 4496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:04.705039 4496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:04.708328 4496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b306b130a91347ffb40e17db0df0669d: Sending DeleteTablet for 3 replicas of tablet d43c7b00a5494523908638338e12778a
I20260619 13:14:04.709333 4884 tablet_service.cc:1514] Processing DeleteTablet for tablet d43c7b00a5494523908638338e12778a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46338
I20260619 13:14:04.709306 4619 tablet_service.cc:1514] Processing DeleteTablet for tablet d43c7b00a5494523908638338e12778a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36512
I20260619 13:14:04.709444 4752 tablet_service.cc:1514] Processing DeleteTablet for tablet d43c7b00a5494523908638338e12778a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52330
I20260619 13:14:04.710161 4990 tablet_replica.cc:330] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14: stopping tablet replica
I20260619 13:14:04.710634 4990 raft_consensus.cc:2229] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:04.711184 4990 raft_consensus.cc:2258] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:04.711278 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 4557
I20260619 13:14:04.712107 4991 tablet_replica.cc:330] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863: stopping tablet replica
I20260619 13:14:04.712479 4991 raft_consensus.cc:2229] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:14:04.713140 4991 raft_consensus.cc:2258] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:04.714736 4992 tablet_replica.cc:330] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a: stopping tablet replica
I20260619 13:14:04.715171 4991 ts_tablet_manager.cc:1905] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:04.715143 4992 raft_consensus.cc:2229] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:04.715612 4992 raft_consensus.cc:2258] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:04.715646 4990 ts_tablet_manager.cc:1905] T d43c7b00a5494523908638338e12778a P 2b709df8d36d44d79b35e3aa481b8c14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:04.717370 4992 ts_tablet_manager.cc:1905] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 13:14:04 dist-test-slave-mkt3 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:14:04.723162 4991 ts_tablet_manager.cc:1918] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:04.723361 4991 log.cc:1196] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/d43c7b00a5494523908638338e12778a
I20260619 13:14:04.723837 4992 ts_tablet_manager.cc:1918] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:04.724051 4992 log.cc:1196] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/d43c7b00a5494523908638338e12778a
I20260619 13:14:04.724325 4991 ts_tablet_manager.cc:1939] T d43c7b00a5494523908638338e12778a P d196eb606f204717978c729f9be43863: Deleting consensus metadata
I20260619 13:14:04.724584 4992 ts_tablet_manager.cc:1939] T d43c7b00a5494523908638338e12778a P 4887ccc5e56f49fbb5e51899feb0b78a: Deleting consensus metadata
I20260619 13:14:04.726405 4481 catalog_manager.cc:4841] TS 4887ccc5e56f49fbb5e51899feb0b78a (127.0.97.2:43649): tablet d43c7b00a5494523908638338e12778a (table test-table [id=f535315e1a7d4b2ea65e476bdf09e866]) successfully deleted
I20260619 13:14:04.726759 4481 catalog_manager.cc:4841] TS d196eb606f204717978c729f9be43863 (127.0.97.3:43157): tablet d43c7b00a5494523908638338e12778a (table test-table [id=f535315e1a7d4b2ea65e476bdf09e866]) successfully deleted
I20260619 13:14:04.729318 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.712500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.0.97.1:32923)
0619 13:14:04.712655 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:04.712661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:04.712678 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:14:04.712856 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:04.712861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:04.712934 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:04.713011 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:04.714715 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.715803 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.717474 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.718063 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.718169 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.722898 (+ 4729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:04.722912 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:04.722928 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:04.722966 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:04.726531 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:04.727330 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:04.727339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:04.727343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:04.727492 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:04.727803 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:04.727808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:04.727812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:04.728278 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:04.728561 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:04.728743 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 13:14:04.728963 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:14:04.729926 4679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:04.815012 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 4557
I20260619 13:14:04.829073 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 4690
W20260619 13:14:04.829133 4481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:32923 (error 108)
I20260619 13:14:04.831174 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.830698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32923 (local address 127.0.0.1:36536)
0619 13:14:04.830868 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:04.830983 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 13:14:04.831421 4481 catalog_manager.cc:4584] TS 2b709df8d36d44d79b35e3aa481b8c14 (127.0.97.1:32923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d43c7b00a5494523908638338e12778a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32923: connect: Connection refused (error 111)
Jun 19 13:14:04 dist-test-slave-mkt3 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:14:04.843299 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.830198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52336 (local address 127.0.97.2:43649)
0619 13:14:04.830361 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:04.830365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:04.830381 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:14:04.830433 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:04.830436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:04.830507 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:04.830605 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:04.832025 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.832740 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.834171 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.834541 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.834651 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.838255 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:04.838268 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:04.838281 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:04.838314 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:04.840795 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:04.841406 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:04.841414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:04.841416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:04.841541 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:04.841898 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:04.841902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:04.841904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:04.842257 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:04.842514 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:04.842809 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 13:14:04.843010 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:14:04.843967 4812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:04.884963 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.884465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32923 (local address 127.0.0.1:36550)
0619 13:14:04.884685 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:04.884796 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 13:14:04.886106 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.885704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32923 (local address 127.0.0.1:36558)
0619 13:14:04.885875 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:04.885961 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 13:14:04.946830 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.946298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32923 (local address 127.0.0.1:36572)
0619 13:14:04.946522 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:04.946643 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 13:14:04.947593 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 4690
I20260619 13:14:04.954874 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.954377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32923 (local address 127.0.0.1:36574)
0619 13:14:04.954638 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:04.954733 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260619 13:14:04.966512 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 4822
Jun 19 13:14:04 dist-test-slave-mkt3 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:14:04.983238 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:04.967331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46350 (local address 127.0.97.3:43157)
0619 13:14:04.967739 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:04.967745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:04.967763 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:14:04.967826 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:04.967832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:04.967911 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:04.968015 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:04.969585 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.970784 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.972249 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:04.972738 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:04.972862 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:04.977195 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:04.977207 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:04.977222 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:04.977270 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:04.980201 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:04.981062 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:04.981072 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:04.981076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:04.981255 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:04.981599 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:04.981604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:04.981608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:04.982069 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:04.982365 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:04.982622 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 13:14:04.982816 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260619 13:14:04.983882 4944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:05.035370 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:05.034912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32923 (local address 127.0.0.1:36588)
0619 13:14:05.035129 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:05.035223 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 13:14:05.036456 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:05.036069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32923 (local address 127.0.0.1:36590)
0619 13:14:05.036276 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:05.036344 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 13:14:05.105320 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 4822
I20260619 13:14:05.120282 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 4465
I20260619 13:14:05.121322 4526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:05.226739 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 4465
I20260619 13:14:05.226847 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:05.226155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32923 (local address 127.0.0.1:36596)
0619 13:14:05.226554 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:05.226650 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:32923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261}
2026-06-19T13:14:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3909 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T13:14:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:14:05Z Disabled control of system clock
I20260619 13:14:05.283955 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34159
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 13:14:05.581795 5002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:05.582077 5002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:05.582204 5002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:05.589154 5002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:05.589259 5002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:05.589315 5002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:05.589390 5002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:05.589447 5002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:14:05.598286 5002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34159
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:05.599442 5002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:05.601145 5002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:05.611896 5007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:05.611959 5010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:05.611930 5008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:05.612792 5002 server_base.cc:1034] running on GCE node
I20260619 13:14:05.613495 5002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:05.614797 5002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:05.615999 5002 hybrid_clock.cc:648] HybridClock initialized: now 1781874845615949 us; error 41 us; skew 500 ppm
I20260619 13:14:05.616302 5002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:05.618371 5002 webserver.cc:460] Webserver started at http://127.0.97.62:38501/ using document root <none> and password file <none>
I20260619 13:14:05.619028 5002 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:05.619123 5002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:05.619410 5002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:05.622133 5002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "6b69bd2a3b0948cb82e447407199a1f5"
format_stamp: "Formatted at 2026-06-19 13:14:05 on dist-test-slave-mkt3"
server_key: "02efd9122e48d34e052bb7e651583ae2"
server_key_iv: "4a379f3b6302007819d1545c45505a94"
server_key_version: "encryptionkey@0"
I20260619 13:14:05.622846 5002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "6b69bd2a3b0948cb82e447407199a1f5"
format_stamp: "Formatted at 2026-06-19 13:14:05 on dist-test-slave-mkt3"
server_key: "02efd9122e48d34e052bb7e651583ae2"
server_key_iv: "4a379f3b6302007819d1545c45505a94"
server_key_version: "encryptionkey@0"
I20260619 13:14:05.627420 5002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:14:05.630589 5016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:05.631800 5002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:14:05.631946 5002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "6b69bd2a3b0948cb82e447407199a1f5"
format_stamp: "Formatted at 2026-06-19 13:14:05 on dist-test-slave-mkt3"
server_key: "02efd9122e48d34e052bb7e651583ae2"
server_key_iv: "4a379f3b6302007819d1545c45505a94"
server_key_version: "encryptionkey@0"
I20260619 13:14:05.632125 5002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:05.637339 5002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:05.638051 5002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:05.638300 5002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:05.654265 5002 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34159
I20260619 13:14:05.654317 5067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34159 every 8 connection(s)
I20260619 13:14:05.655521 5002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:14:05.658965 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 5002
I20260619 13:14:05.659102 5068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:05.659135 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:14:05.659726 388 external_mini_cluster.cc:1428] Setting key 28c5f3380462f9642f019dcc7b7210c8
I20260619 13:14:05.668751 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:05.661580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.0.97.62:34159)
0619 13:14:05.662691 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:05.662705 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:05.662757 (+ 52us) server_negotiation.cc:407] Connection header received
0619 13:14:05.665084 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:05.665110 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:05.665635 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:05.666083 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:05.666974 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:05.666982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:05.666997 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:05.667011 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:05.667076 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:05.667095 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:05.667292 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:05.667405 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 13:14:05.667652 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":357,"threads_started":1}
I20260619 13:14:05.671891 5068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5: Bootstrap starting.
I20260619 13:14:05.674595 5068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:05.675452 5068 log.cc:824] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:05.677752 5068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5: No bootstrap required, opened a new log
I20260619 13:14:05.684229 5068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b69bd2a3b0948cb82e447407199a1f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34159 } }
I20260619 13:14:05.684520 5068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:05.684607 5068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b69bd2a3b0948cb82e447407199a1f5, State: Initialized, Role: FOLLOWER
I20260619 13:14:05.685101 5068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b69bd2a3b0948cb82e447407199a1f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34159 } }
I20260619 13:14:05.685302 5068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:14:05.685421 5068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:14:05.685556 5068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:05.687544 5068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b69bd2a3b0948cb82e447407199a1f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34159 } }
I20260619 13:14:05.687980 5068 leader_election.cc:302] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b69bd2a3b0948cb82e447407199a1f5; no voters:
I20260619 13:14:05.688477 5068 leader_election.cc:288] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:14:05.688573 5073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:05.689218 5073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [term 1 LEADER]: Becoming Leader. State: Replica: 6b69bd2a3b0948cb82e447407199a1f5, State: Running, Role: LEADER
I20260619 13:14:05.689700 5073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b69bd2a3b0948cb82e447407199a1f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34159 } }
I20260619 13:14:05.690284 5068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:05.693600 5075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b69bd2a3b0948cb82e447407199a1f5. Latest consensus state: current_term: 1 leader_uuid: "6b69bd2a3b0948cb82e447407199a1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b69bd2a3b0948cb82e447407199a1f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34159 } } }
I20260619 13:14:05.693981 5075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:05.695431 5081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:14:05.694038 5074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b69bd2a3b0948cb82e447407199a1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b69bd2a3b0948cb82e447407199a1f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34159 } } }
I20260619 13:14:05.695642 5074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:05.699945 5081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:14:05.708212 5081 catalog_manager.cc:1347] Generated new cluster ID: ca6baa34306a4378b89c57480dab061b
I20260619 13:14:05.708302 5081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:14:05.720077 5081 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:14:05.720430 5081 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:14:05.728121 5081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5: Generated new TSK 0
I20260619 13:14:05.728796 5081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:14:05.738147 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34159
--builtin_ntp_servers=127.0.97.20:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 13:14:06.055440 5092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:06.055692 5092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:06.055802 5092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:06.062826 5092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:06.062917 5092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:06.063043 5092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:14:06.071786 5092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:06.072898 5092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:06.074545 5092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:06.086514 5097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:06.086612 5100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:06.086634 5098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:06.087433 5092 server_base.cc:1034] running on GCE node
I20260619 13:14:06.087983 5092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:06.088800 5092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:06.089972 5092 hybrid_clock.cc:648] HybridClock initialized: now 1781874846089919 us; error 48 us; skew 500 ppm
I20260619 13:14:06.090276 5092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:06.092278 5092 webserver.cc:460] Webserver started at http://127.0.97.1:33745/ using document root <none> and password file <none>
I20260619 13:14:06.092878 5092 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:06.092968 5092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:06.093256 5092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:06.095954 5092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "c2d558ba4dbb4a4dad97610e6999bf00"
format_stamp: "Formatted at 2026-06-19 13:14:06 on dist-test-slave-mkt3"
server_key: "15c816e39fe051d424e857ea8475b95e"
server_key_iv: "2c9331ba886d00bece14c0b0c75df3d2"
server_key_version: "encryptionkey@0"
I20260619 13:14:06.096669 5092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c2d558ba4dbb4a4dad97610e6999bf00"
format_stamp: "Formatted at 2026-06-19 13:14:06 on dist-test-slave-mkt3"
server_key: "15c816e39fe051d424e857ea8475b95e"
server_key_iv: "2c9331ba886d00bece14c0b0c75df3d2"
server_key_version: "encryptionkey@0"
I20260619 13:14:06.102358 5092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 13:14:06.106783 5106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:06.108039 5092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260619 13:14:06.108180 5092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "c2d558ba4dbb4a4dad97610e6999bf00"
format_stamp: "Formatted at 2026-06-19 13:14:06 on dist-test-slave-mkt3"
server_key: "15c816e39fe051d424e857ea8475b95e"
server_key_iv: "2c9331ba886d00bece14c0b0c75df3d2"
server_key_version: "encryptionkey@0"
I20260619 13:14:06.108386 5092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:06.113777 5092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:06.114603 5092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:06.114833 5092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:06.115684 5092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:06.116943 5092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:06.117033 5092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:06.117138 5092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:06.117210 5092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:06.145017 5092 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37541
I20260619 13:14:06.145080 5218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37541 every 8 connection(s)
I20260619 13:14:06.146343 5092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:14:06.153493 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:06.148586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56221 (local address 127.0.97.62:34159)
0619 13:14:06.148780 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:06.148786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:06.150170 (+ 1384us) server_negotiation.cc:407] Connection header received
0619 13:14:06.151721 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:06.151725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:06.151776 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:06.151859 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:06.152961 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:06.152965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:06.152968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:06.152982 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:06.153012 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:06.153023 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:06.153128 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:06.153250 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 13:14:06.153295 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:14:06.154386 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:06.149146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34159 (local address 127.0.97.1:56221)
0619 13:14:06.150032 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:06.150076 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:06.151489 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:06.152057 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:06.152065 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:06.152153 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:06.152711 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:14:06.152713 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:06.152748 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:14:06.152819 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:06.153095 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:06.153101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:06.153106 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:14:06.153306 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 13:14:06.153484 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":247,"threads_started":1}
I20260619 13:14:06.155619 5219 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34159
I20260619 13:14:06.155857 5219 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:06.156369 5219 heartbeater.cc:511] Master 127.0.97.62:34159 requested a full tablet report, sending...
I20260619 13:14:06.156425 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 5092
I20260619 13:14:06.156589 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:14:06.157219 388 external_mini_cluster.cc:1428] Setting key 3fe23cc9b5ca7bfe0ec27dc0ae5f9374
I20260619 13:14:06.158470 5033 ts_manager.cc:194] Registered new tserver with Master: c2d558ba4dbb4a4dad97610e6999bf00 (127.0.97.1:37541)
I20260619 13:14:06.160279 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34159
--builtin_ntp_servers=127.0.97.20:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 13:14:06.457456 5222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:06.457748 5222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:06.457894 5222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:06.464916 5222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:06.465003 5222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:06.465149 5222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:14:06.473857 5222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:06.475063 5222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:06.476765 5222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:06.488772 5227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:06.488804 5228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:06.488739 5230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:06.489542 5222 server_base.cc:1034] running on GCE node
I20260619 13:14:06.490123 5222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:06.491063 5222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:06.492246 5222 hybrid_clock.cc:648] HybridClock initialized: now 1781874846492180 us; error 60 us; skew 500 ppm
I20260619 13:14:06.492538 5222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:06.494673 5222 webserver.cc:460] Webserver started at http://127.0.97.2:35475/ using document root <none> and password file <none>
I20260619 13:14:06.495314 5222 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:06.495398 5222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:06.495689 5222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:06.498492 5222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "f3fd00136b584e338aac1f79404ea75e"
format_stamp: "Formatted at 2026-06-19 13:14:06 on dist-test-slave-mkt3"
server_key: "03c9cb73f61a1799396651619457b92b"
server_key_iv: "704463d7f6fa796fa4c4ff6be5a756ce"
server_key_version: "encryptionkey@0"
I20260619 13:14:06.499255 5222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f3fd00136b584e338aac1f79404ea75e"
format_stamp: "Formatted at 2026-06-19 13:14:06 on dist-test-slave-mkt3"
server_key: "03c9cb73f61a1799396651619457b92b"
server_key_iv: "704463d7f6fa796fa4c4ff6be5a756ce"
server_key_version: "encryptionkey@0"
I20260619 13:14:06.504168 5222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:14:06.507560 5236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:06.508927 5222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:14:06.509061 5222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "f3fd00136b584e338aac1f79404ea75e"
format_stamp: "Formatted at 2026-06-19 13:14:06 on dist-test-slave-mkt3"
server_key: "03c9cb73f61a1799396651619457b92b"
server_key_iv: "704463d7f6fa796fa4c4ff6be5a756ce"
server_key_version: "encryptionkey@0"
I20260619 13:14:06.509262 5222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:06.514926 5222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:06.515647 5222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:06.515874 5222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:06.516705 5222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:06.518021 5222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:06.518102 5222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:06.518203 5222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:06.518265 5222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:06.546044 5222 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36795
I20260619 13:14:06.546079 5348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36795 every 8 connection(s)
I20260619 13:14:06.547228 5222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:14:06.554301 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:06.549514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45539 (local address 127.0.97.62:34159)
0619 13:14:06.549683 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:06.549688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:06.550996 (+ 1308us) server_negotiation.cc:407] Connection header received
0619 13:14:06.552526 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:06.552530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:06.552579 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:06.552669 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:06.553726 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:06.553730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:06.553733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:06.553747 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:06.553776 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:06.553788 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:06.553908 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:06.554046 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 13:14:06.554090 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:14:06.555132 5351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:06.550038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34159 (local address 127.0.97.2:45539)
0619 13:14:06.550882 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:06.550915 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:06.552314 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:06.552833 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:06.552840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:06.552923 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:06.553478 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:14:06.553480 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:06.553514 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:14:06.553590 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:06.553890 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:06.553898 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:06.553905 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 13:14:06.554121 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 13:14:06.554303 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":253,"threads_started":1}
I20260619 13:14:06.555302 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 5222
I20260619 13:14:06.555434 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:14:06.555872 388 external_mini_cluster.cc:1428] Setting key 29e3e159dc303db3134c7b4bbe7d9301
I20260619 13:14:06.556348 5349 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34159
I20260619 13:14:06.556591 5349 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:06.557679 5349 heartbeater.cc:511] Master 127.0.97.62:34159 requested a full tablet report, sending...
I20260619 13:14:06.558495 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34159
--builtin_ntp_servers=127.0.97.20:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 13:14:06.559120 5033 ts_manager.cc:194] Registered new tserver with Master: f3fd00136b584e338aac1f79404ea75e (127.0.97.2:36795)
W20260619 13:14:06.858016 5352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:06.858280 5352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:06.858362 5352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:06.865260 5352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:06.865345 5352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:06.865456 5352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:14:06.874091 5352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:06.875227 5352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:06.876873 5352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:06.888803 5357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:06.888746 5358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:06.888787 5360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:06.889523 5352 server_base.cc:1034] running on GCE node
I20260619 13:14:06.890141 5352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:06.891026 5352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:06.892208 5352 hybrid_clock.cc:648] HybridClock initialized: now 1781874846892155 us; error 53 us; skew 500 ppm
I20260619 13:14:06.892489 5352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:06.894565 5352 webserver.cc:460] Webserver started at http://127.0.97.3:36495/ using document root <none> and password file <none>
I20260619 13:14:06.895181 5352 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:06.895275 5352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:06.895550 5352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:06.898269 5352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "5771fde4ee2a44ed9ad31b62875ca5be"
format_stamp: "Formatted at 2026-06-19 13:14:06 on dist-test-slave-mkt3"
server_key: "c7420a859e3fbf1eda25754c5f5ccb14"
server_key_iv: "07cb0341624705b3b24eabf0b0c4463a"
server_key_version: "encryptionkey@0"
I20260619 13:14:06.899000 5352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5771fde4ee2a44ed9ad31b62875ca5be"
format_stamp: "Formatted at 2026-06-19 13:14:06 on dist-test-slave-mkt3"
server_key: "c7420a859e3fbf1eda25754c5f5ccb14"
server_key_iv: "07cb0341624705b3b24eabf0b0c4463a"
server_key_version: "encryptionkey@0"
I20260619 13:14:06.904505 5352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:14:06.908043 5366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:06.909301 5352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:14:06.909441 5352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "5771fde4ee2a44ed9ad31b62875ca5be"
format_stamp: "Formatted at 2026-06-19 13:14:06 on dist-test-slave-mkt3"
server_key: "c7420a859e3fbf1eda25754c5f5ccb14"
server_key_iv: "07cb0341624705b3b24eabf0b0c4463a"
server_key_version: "encryptionkey@0"
I20260619 13:14:06.909641 5352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:06.914953 5352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:06.915691 5352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:06.915925 5352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:06.916761 5352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:06.918157 5352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:06.918255 5352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:06.918365 5352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:06.918432 5352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:06.946604 5352 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39085
I20260619 13:14:06.946648 5478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39085 every 8 connection(s)
I20260619 13:14:06.948050 5352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:14:06.954813 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 5352
I20260619 13:14:06.954980 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:14:06.955492 388 external_mini_cluster.cc:1428] Setting key ed6820afb4159534f00f5f667576e13e
I20260619 13:14:06.957381 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:06.950841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35793 (local address 127.0.97.62:34159)
0619 13:14:06.951044 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:06.951050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:06.953152 (+ 2102us) server_negotiation.cc:407] Connection header received
0619 13:14:06.955168 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:06.955172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:06.955226 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:06.955308 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:06.956720 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:06.956725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:06.956730 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:06.956750 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:06.956785 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:06.956799 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:06.956902 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:06.957087 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 13:14:06.957134 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 13:14:06.958580 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:06.951546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34159 (local address 127.0.97.3:35793)
0619 13:14:06.952976 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:06.953029 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:06.954884 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:06.955537 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:06.955549 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:06.955667 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:06.956388 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:14:06.956391 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:06.956438 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:14:06.956542 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:06.956891 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:06.956901 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:06.956908 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 13:14:06.957187 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 13:14:06.957419 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":633,"threads_started":1}
I20260619 13:14:06.959736 5479 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34159
I20260619 13:14:06.959968 5479 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:06.960469 5479 heartbeater.cc:511] Master 127.0.97.62:34159 requested a full tablet report, sending...
I20260619 13:14:06.961704 5033 ts_manager.cc:194] Registered new tserver with Master: 5771fde4ee2a44ed9ad31b62875ca5be (127.0.97.3:39085)
I20260619 13:14:06.970544 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:14:06.976894 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:06.974653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.0.97.62:34159)
0619 13:14:06.974792 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:06.974796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:06.975245 (+ 449us) server_negotiation.cc:407] Connection header received
0619 13:14:06.975400 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:06.975404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:06.975450 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:06.975527 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:06.976334 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:06.976340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:06.976345 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:06.976361 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:06.976388 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:06.976400 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:06.976496 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:06.976581 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 13:14:06.976644 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 13:14:06.980312 5033 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:14:06.986765 5033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:14:07.013343 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:06.998298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.0.97.3:39085)
0619 13:14:07.010067 (+ 11769us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:07.010076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:07.010112 (+ 36us) server_negotiation.cc:407] Connection header received
0619 13:14:07.010180 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:07.010187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:07.010455 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:07.010618 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:07.012161 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:07.012169 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:07.012184 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:07.012200 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:07.012254 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:07.012278 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:07.012431 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:07.012840 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 13:14:07.012954 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11612,"thread_start_us":11451,"threads_started":1}
I20260619 13:14:07.013717 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:06.997802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33960 (local address 127.0.97.2:36795)
0619 13:14:07.010363 (+ 12561us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:07.010370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:07.010400 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:14:07.010459 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:07.010466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:07.010647 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:07.010802 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:07.012616 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:07.012635 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:07.012651 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:07.012683 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:07.012790 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:07.012812 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:07.012976 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:07.013188 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 13:14:07.013286 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12408,"thread_start_us":13673,"threads_started":1}
I20260619 13:14:07.014957 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:06.997655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39085 (local address 127.0.0.1:52308)
0619 13:14:07.008771 (+ 11116us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:07.008786 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:07.008872 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:07.010687 (+ 1815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:07.010696 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:07.010813 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:07.011602 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:14:07.011606 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:07.011651 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:14:07.011738 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:07.012479 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:07.012490 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:07.012498 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 13:14:07.014325 (+ 1827us) client_negotiation.cc:240] Negotiation successful
0619 13:14:07.014419 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11006,"thread_start_us":10864,"threads_started":1}
I20260619 13:14:07.015257 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:06.997642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36795 (local address 127.0.0.1:33960)
0619 13:14:07.008141 (+ 10499us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:07.008189 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:07.008327 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:07.011729 (+ 3402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:07.011734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:07.011757 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:07.012411 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:14:07.012414 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:07.012428 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:14:07.012467 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:07.013032 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:07.013037 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:07.013043 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 13:14:07.014496 (+ 1453us) client_negotiation.cc:240] Negotiation successful
0619 13:14:07.014556 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10225,"thread_start_us":10079,"threads_started":1}
I20260619 13:14:07.018715 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.012183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37541 (local address 127.0.0.1:51828)
0619 13:14:07.015167 (+ 2984us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:07.015185 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:07.015269 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:07.016381 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:07.016386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:07.016418 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:07.017983 (+ 1565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:14:07.017986 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:07.018000 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:14:07.018038 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:07.018339 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:07.018344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:07.018348 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 13:14:07.018436 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 13:14:07.018483 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2892,"thread_start_us":3692,"threads_started":1}
I20260619 13:14:07.018817 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.009192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.0.97.1:37541)
0619 13:14:07.015885 (+ 6693us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:07.015894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:07.015921 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:14:07.015982 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:07.015989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:07.016219 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:07.016371 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:07.018164 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:07.018170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:07.018180 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:07.018194 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:07.018240 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:07.018253 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:07.018362 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:07.018446 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 13:14:07.018535 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6530,"thread_start_us":6392,"threads_started":1}
I20260619 13:14:07.019604 5284 tablet_service.cc:1467] Processing CreateTablet for tablet f5e3ad80b56a49958216cfa3d6fa58c9 (DEFAULT_TABLE table=test-table [id=cfc6e1fecd4540c2b6c22d736d34dc18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:07.019604 5414 tablet_service.cc:1467] Processing CreateTablet for tablet f5e3ad80b56a49958216cfa3d6fa58c9 (DEFAULT_TABLE table=test-table [id=cfc6e1fecd4540c2b6c22d736d34dc18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:07.021051 5284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e3ad80b56a49958216cfa3d6fa58c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:07.021139 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e3ad80b56a49958216cfa3d6fa58c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:07.021920 5154 tablet_service.cc:1467] Processing CreateTablet for tablet f5e3ad80b56a49958216cfa3d6fa58c9 (DEFAULT_TABLE table=test-table [id=cfc6e1fecd4540c2b6c22d736d34dc18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:07.023015 5154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e3ad80b56a49958216cfa3d6fa58c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:07.033450 5497 tablet_bootstrap.cc:492] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00: Bootstrap starting.
I20260619 13:14:07.036136 5498 tablet_bootstrap.cc:492] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e: Bootstrap starting.
I20260619 13:14:07.037966 5499 tablet_bootstrap.cc:492] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be: Bootstrap starting.
I20260619 13:14:07.038687 5497 tablet_bootstrap.cc:654] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:07.039487 5498 tablet_bootstrap.cc:654] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:07.039909 5497 log.cc:824] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:07.040620 5498 log.cc:824] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:07.040628 5499 tablet_bootstrap.cc:654] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:07.041595 5499 log.cc:824] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:07.042732 5497 tablet_bootstrap.cc:492] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00: No bootstrap required, opened a new log
I20260619 13:14:07.042987 5497 ts_tablet_manager.cc:1397] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 13:14:07.043726 5498 tablet_bootstrap.cc:492] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e: No bootstrap required, opened a new log
I20260619 13:14:07.043939 5498 ts_tablet_manager.cc:1397] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260619 13:14:07.044302 5499 tablet_bootstrap.cc:492] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be: No bootstrap required, opened a new log
I20260619 13:14:07.044528 5499 ts_tablet_manager.cc:1397] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260619 13:14:07.050853 5498 raft_consensus.cc:348] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fd00136b584e338aac1f79404ea75e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36795 } } peers { permanent_uuid: "5771fde4ee2a44ed9ad31b62875ca5be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39085 } } peers { permanent_uuid: "c2d558ba4dbb4a4dad97610e6999bf00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37541 } }
I20260619 13:14:07.051187 5498 raft_consensus.cc:374] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:07.051270 5498 raft_consensus.cc:729] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3fd00136b584e338aac1f79404ea75e, State: Initialized, Role: FOLLOWER
I20260619 13:14:07.051800 5498 consensus_queue.cc:260] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fd00136b584e338aac1f79404ea75e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36795 } } peers { permanent_uuid: "5771fde4ee2a44ed9ad31b62875ca5be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39085 } } peers { permanent_uuid: "c2d558ba4dbb4a4dad97610e6999bf00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37541 } }
I20260619 13:14:07.052799 5349 heartbeater.cc:503] Master 127.0.97.62:34159 was elected leader, sending a full tablet report...
I20260619 13:14:07.052754 5497 raft_consensus.cc:348] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fd00136b584e338aac1f79404ea75e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36795 } } peers { permanent_uuid: "5771fde4ee2a44ed9ad31b62875ca5be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39085 } } peers { permanent_uuid: "c2d558ba4dbb4a4dad97610e6999bf00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37541 } }
I20260619 13:14:07.053171 5497 raft_consensus.cc:374] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:07.053246 5498 ts_tablet_manager.cc:1428] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260619 13:14:07.053263 5497 raft_consensus.cc:729] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d558ba4dbb4a4dad97610e6999bf00, State: Initialized, Role: FOLLOWER
I20260619 13:14:07.053884 5497 consensus_queue.cc:260] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fd00136b584e338aac1f79404ea75e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36795 } } peers { permanent_uuid: "5771fde4ee2a44ed9ad31b62875ca5be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39085 } } peers { permanent_uuid: "c2d558ba4dbb4a4dad97610e6999bf00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37541 } }
I20260619 13:14:07.054862 5219 heartbeater.cc:503] Master 127.0.97.62:34159 was elected leader, sending a full tablet report...
I20260619 13:14:07.054867 5499 raft_consensus.cc:348] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fd00136b584e338aac1f79404ea75e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36795 } } peers { permanent_uuid: "5771fde4ee2a44ed9ad31b62875ca5be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39085 } } peers { permanent_uuid: "c2d558ba4dbb4a4dad97610e6999bf00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37541 } }
I20260619 13:14:07.055250 5499 raft_consensus.cc:374] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:07.055332 5499 raft_consensus.cc:729] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5771fde4ee2a44ed9ad31b62875ca5be, State: Initialized, Role: FOLLOWER
I20260619 13:14:07.055459 5497 ts_tablet_manager.cc:1428] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260619 13:14:07.056085 5499 consensus_queue.cc:260] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fd00136b584e338aac1f79404ea75e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36795 } } peers { permanent_uuid: "5771fde4ee2a44ed9ad31b62875ca5be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39085 } } peers { permanent_uuid: "c2d558ba4dbb4a4dad97610e6999bf00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37541 } }
I20260619 13:14:07.057240 5479 heartbeater.cc:503] Master 127.0.97.62:34159 was elected leader, sending a full tablet report...
I20260619 13:14:07.057811 5499 ts_tablet_manager.cc:1428] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 13:14:07.086760 5504 raft_consensus.cc:482] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:14:07.086994 5504 raft_consensus.cc:504] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fd00136b584e338aac1f79404ea75e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36795 } } peers { permanent_uuid: "5771fde4ee2a44ed9ad31b62875ca5be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39085 } } peers { permanent_uuid: "c2d558ba4dbb4a4dad97610e6999bf00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37541 } }
I20260619 13:14:07.088516 5504 leader_election.cc:288] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3fd00136b584e338aac1f79404ea75e (127.0.97.2:36795), 5771fde4ee2a44ed9ad31b62875ca5be (127.0.97.3:39085)
I20260619 13:14:07.091516 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.088791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41055 (local address 127.0.97.2:36795)
0619 13:14:07.088956 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:07.088963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:07.089783 (+ 820us) server_negotiation.cc:407] Connection header received
0619 13:14:07.089910 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:07.089915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:07.089967 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:07.090045 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:07.090957 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:07.090963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:07.090968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:07.090984 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:07.091011 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:07.091022 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:07.091123 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:07.091221 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 13:14:07.091280 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:14:07.091668 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.089052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57685 (local address 127.0.97.3:39085)
0619 13:14:07.089240 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:07.089247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:07.089566 (+ 319us) server_negotiation.cc:407] Connection header received
0619 13:14:07.089724 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:07.089728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:07.089781 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:07.089886 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:07.091088 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:07.091092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:07.091095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:07.091109 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:07.091139 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:07.091149 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:07.091249 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:07.091372 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 13:14:07.091428 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:14:07.091879 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.088756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39085 (local address 127.0.97.1:57685)
0619 13:14:07.089462 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:07.089477 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:07.089601 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:07.090183 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:07.090186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:07.090207 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:07.090919 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:14:07.090921 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:07.090933 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:14:07.090965 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:07.091232 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:07.091236 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:07.091241 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:14:07.091407 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 13:14:07.091453 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":404,"threads_started":1}
I20260619 13:14:07.092041 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.088832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36795 (local address 127.0.97.1:41055)
0619 13:14:07.089693 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:07.089707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:07.089795 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:07.090077 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:07.090080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:07.090121 (+ 41us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:07.090779 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:14:07.090781 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:07.090793 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:14:07.090826 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:07.091099 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:07.091103 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:07.091108 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:14:07.091258 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 13:14:07.091302 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":578,"threads_started":1}
I20260619 13:14:07.092847 5304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e3ad80b56a49958216cfa3d6fa58c9" candidate_uuid: "c2d558ba4dbb4a4dad97610e6999bf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3fd00136b584e338aac1f79404ea75e" is_pre_election: true
I20260619 13:14:07.092849 5434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e3ad80b56a49958216cfa3d6fa58c9" candidate_uuid: "c2d558ba4dbb4a4dad97610e6999bf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5771fde4ee2a44ed9ad31b62875ca5be" is_pre_election: true
I20260619 13:14:07.093300 5304 raft_consensus.cc:2454] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2d558ba4dbb4a4dad97610e6999bf00 in term 0.
I20260619 13:14:07.093300 5434 raft_consensus.cc:2454] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2d558ba4dbb4a4dad97610e6999bf00 in term 0.
I20260619 13:14:07.093890 5108 leader_election.cc:302] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2d558ba4dbb4a4dad97610e6999bf00, f3fd00136b584e338aac1f79404ea75e; no voters:
I20260619 13:14:07.094241 5504 raft_consensus.cc:2781] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:14:07.094374 5504 raft_consensus.cc:482] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:14:07.094473 5504 raft_consensus.cc:3037] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:07.096518 5504 raft_consensus.cc:504] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fd00136b584e338aac1f79404ea75e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36795 } } peers { permanent_uuid: "5771fde4ee2a44ed9ad31b62875ca5be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39085 } } peers { permanent_uuid: "c2d558ba4dbb4a4dad97610e6999bf00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37541 } }
I20260619 13:14:07.097064 5504 leader_election.cc:288] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [CANDIDATE]: Term 1 election: Requested vote from peers f3fd00136b584e338aac1f79404ea75e (127.0.97.2:36795), 5771fde4ee2a44ed9ad31b62875ca5be (127.0.97.3:39085)
I20260619 13:14:07.097332 5304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e3ad80b56a49958216cfa3d6fa58c9" candidate_uuid: "c2d558ba4dbb4a4dad97610e6999bf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3fd00136b584e338aac1f79404ea75e"
I20260619 13:14:07.097486 5304 raft_consensus.cc:3037] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:07.097666 5434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e3ad80b56a49958216cfa3d6fa58c9" candidate_uuid: "c2d558ba4dbb4a4dad97610e6999bf00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5771fde4ee2a44ed9ad31b62875ca5be"
I20260619 13:14:07.097827 5434 raft_consensus.cc:3037] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:07.099651 5304 raft_consensus.cc:2454] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2d558ba4dbb4a4dad97610e6999bf00 in term 1.
I20260619 13:14:07.099957 5108 leader_election.cc:302] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2d558ba4dbb4a4dad97610e6999bf00, f3fd00136b584e338aac1f79404ea75e; no voters:
I20260619 13:14:07.100059 5434 raft_consensus.cc:2454] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2d558ba4dbb4a4dad97610e6999bf00 in term 1.
I20260619 13:14:07.100173 5504 raft_consensus.cc:2781] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:07.100589 5504 raft_consensus.cc:686] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [term 1 LEADER]: Becoming Leader. State: Replica: c2d558ba4dbb4a4dad97610e6999bf00, State: Running, Role: LEADER
I20260619 13:14:07.101022 5504 consensus_queue.cc:237] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fd00136b584e338aac1f79404ea75e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36795 } } peers { permanent_uuid: "5771fde4ee2a44ed9ad31b62875ca5be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39085 } } peers { permanent_uuid: "c2d558ba4dbb4a4dad97610e6999bf00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37541 } }
I20260619 13:14:07.105036 5031 catalog_manager.cc:5495] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2d558ba4dbb4a4dad97610e6999bf00 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c2d558ba4dbb4a4dad97610e6999bf00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fd00136b584e338aac1f79404ea75e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5771fde4ee2a44ed9ad31b62875ca5be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2d558ba4dbb4a4dad97610e6999bf00" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37541 } health_report { overall_health: HEALTHY } } }
I20260619 13:14:07.126617 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.124607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.0.97.1:37541)
0619 13:14:07.124788 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:07.124793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:07.124813 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:14:07.125027 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:07.125031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:07.125113 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:07.125197 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:07.126047 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:07.126053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:07.126058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:07.126076 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:07.126105 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:07.126117 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:07.126208 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:07.126297 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 13:14:07.126351 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:14:07.133949 5434 raft_consensus.cc:1261] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be [term 1 FOLLOWER]: Refusing update from remote peer c2d558ba4dbb4a4dad97610e6999bf00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:07.133950 5304 raft_consensus.cc:1261] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e [term 1 FOLLOWER]: Refusing update from remote peer c2d558ba4dbb4a4dad97610e6999bf00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:07.134775 5508 consensus_queue.cc:1035] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5771fde4ee2a44ed9ad31b62875ca5be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:07.135068 5504 consensus_queue.cc:1035] T f5e3ad80b56a49958216cfa3d6fa58c9 P c2d558ba4dbb4a4dad97610e6999bf00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3fd00136b584e338aac1f79404ea75e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:07.148597 5511 mvcc.cc:205] Tried to move back new op lower bound from 7298559373852041216 to 7298559373729316864. Current Snapshot: MvccSnapshot[applied={T|T < 7298559373852041216 or (T in {7298559373852041216})}]
I20260619 13:14:07.151943 5513 mvcc.cc:205] Tried to move back new op lower bound from 7298559373852041216 to 7298559373729316864. Current Snapshot: MvccSnapshot[applied={T|T < 7298559373852041216}]
I20260619 13:14:07.161629 5031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:07.161968 5031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:07.165822 5031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b69bd2a3b0948cb82e447407199a1f5: Sending DeleteTablet for 3 replicas of tablet f5e3ad80b56a49958216cfa3d6fa58c9
I20260619 13:14:07.167017 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e3ad80b56a49958216cfa3d6fa58c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:07 UTC) from {username='slave'} at 127.0.0.1:52308
I20260619 13:14:07.167037 5154 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e3ad80b56a49958216cfa3d6fa58c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:07 UTC) from {username='slave'} at 127.0.0.1:51828
I20260619 13:14:07.167060 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e3ad80b56a49958216cfa3d6fa58c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:07 UTC) from {username='slave'} at 127.0.0.1:52308
I20260619 13:14:07.167698 5284 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e3ad80b56a49958216cfa3d6fa58c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:07 UTC) from {username='slave'} at 127.0.0.1:33960
I20260619 13:14:07.168761 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 5092
I20260619 13:14:07.182302 5522 tablet_replica.cc:330] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e: stopping tablet replica
I20260619 13:14:07.182818 5522 raft_consensus.cc:2229] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:07.183393 5522 raft_consensus.cc:2258] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:07.185814 5522 ts_tablet_manager.cc:1905] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:07.188973 5520 tablet_replica.cc:330] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be: stopping tablet replica
I20260619 13:14:07.189373 5520 raft_consensus.cc:2229] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:07.190883 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.170254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.0.97.1:37541)
0619 13:14:07.170452 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:07.170459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:07.185899 (+ 15440us) server_negotiation.cc:407] Connection header received
0619 13:14:07.186847 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:07.186853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:07.186914 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:07.186985 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:07.190028 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:07.190034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:07.190040 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:07.190060 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:07.190095 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:07.190111 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:07.190205 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:07.190504 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 13:14:07.190608 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:14:07.191272 5520 raft_consensus.cc:2258] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:07.193288 5522 ts_tablet_manager.cc:1918] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:07.193374 5520 ts_tablet_manager.cc:1905] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:07.193454 5522 log.cc:1196] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/f5e3ad80b56a49958216cfa3d6fa58c9
I20260619 13:14:07.194002 5522 ts_tablet_manager.cc:1939] T f5e3ad80b56a49958216cfa3d6fa58c9 P f3fd00136b584e338aac1f79404ea75e: Deleting consensus metadata
I20260619 13:14:07.195873 5018 catalog_manager.cc:4841] TS f3fd00136b584e338aac1f79404ea75e (127.0.97.2:36795): tablet f5e3ad80b56a49958216cfa3d6fa58c9 (table test-table [id=cfc6e1fecd4540c2b6c22d736d34dc18]) successfully deleted
I20260619 13:14:07.197279 5214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:07.200219 5520 ts_tablet_manager.cc:1918] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:07.200394 5520 log.cc:1196] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/f5e3ad80b56a49958216cfa3d6fa58c9
I20260619 13:14:07.200877 5520 ts_tablet_manager.cc:1939] T f5e3ad80b56a49958216cfa3d6fa58c9 P 5771fde4ee2a44ed9ad31b62875ca5be: Deleting consensus metadata
I20260619 13:14:07.202515 5020 catalog_manager.cc:4841] TS 5771fde4ee2a44ed9ad31b62875ca5be (127.0.97.3:39085): tablet f5e3ad80b56a49958216cfa3d6fa58c9 (table test-table [id=cfc6e1fecd4540c2b6c22d736d34dc18]) successfully deleted
W20260619 13:14:07.202977 5020 catalog_manager.cc:4808] TS 5771fde4ee2a44ed9ad31b62875ca5be (127.0.97.3:39085): delete failed for tablet f5e3ad80b56a49958216cfa3d6fa58c9 because the tablet was not found. No further retry: Not found: Tablet not found: f5e3ad80b56a49958216cfa3d6fa58c9
I20260619 13:14:07.291615 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 5092
I20260619 13:14:07.305612 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 5222
W20260619 13:14:07.305598 5020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37541 (error 108)
I20260619 13:14:07.307305 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.306845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37541 (local address 127.0.0.1:51854)
0619 13:14:07.307008 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:07.307131 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 13:14:07.307533 5020 catalog_manager.cc:4584] TS c2d558ba4dbb4a4dad97610e6999bf00 (127.0.97.1:37541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5e3ad80b56a49958216cfa3d6fa58c9: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37541: connect: Connection refused (error 111)
I20260619 13:14:07.308495 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.306315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.0.97.2:36795)
0619 13:14:07.306480 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:07.306486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:07.306633 (+ 147us) server_negotiation.cc:407] Connection header received
0619 13:14:07.306810 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:07.306816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:07.306911 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:07.307044 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:07.307922 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:07.307929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:07.307934 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:07.307951 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:07.307980 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:07.307991 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:07.308106 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:07.308195 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 13:14:07.308252 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:14:07.309072 5344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:07.361238 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.360767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37541 (local address 127.0.0.1:51870)
0619 13:14:07.360989 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:07.361096 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 13:14:07.362205 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.361894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37541 (local address 127.0.0.1:51878)
0619 13:14:07.362001 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:07.362086 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 13:14:07.392362 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 5222
I20260619 13:14:07.405162 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 5352
I20260619 13:14:07.407836 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.405854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52310 (local address 127.0.97.3:39085)
0619 13:14:07.406000 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:07.406004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:07.406050 (+ 46us) server_negotiation.cc:407] Connection header received
0619 13:14:07.406230 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:07.406234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:07.406326 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:07.406407 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:07.407272 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:07.407275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:07.407278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:07.407288 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:07.407309 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:07.407319 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:07.407412 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:07.407538 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 13:14:07.407581 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 13:14:07.408383 5474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:07.422977 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.422494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37541 (local address 127.0.0.1:51894)
0619 13:14:07.422711 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:07.422815 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 13:14:07.423914 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.423540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37541 (local address 127.0.0.1:51900)
0619 13:14:07.423683 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:07.423763 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 13:14:07.492439 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 5352
I20260619 13:14:07.504347 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.503911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37541 (local address 127.0.0.1:51912)
0619 13:14:07.504085 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:07.504189 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 13:14:07.504729 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 5002
I20260619 13:14:07.505393 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:07.504957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37541 (local address 127.0.0.1:51926)
0619 13:14:07.505147 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:07.505208 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 13:14:07.505743 5063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:07.606062 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 5002
2026-06-19T13:14:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2377 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 13:14:07.641314 388 mini_oidc.cc:152] Starting JWKS server
I20260619 13:14:07.643639 388 webserver.cc:460] Webserver started at https://127.0.0.1:38561/ using document root <none> and password file <none>
I20260619 13:14:07.643894 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 13:14:07.644449 388 webserver.cc:460] Webserver started at http://127.0.0.1:42463/ using document root <none> and password file <none>
2026-06-19T13:14:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:14:07Z Disabled control of system clock
I20260619 13:14:07.666805 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38475
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38475
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38561/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/testchainca.cert with env {}
W20260619 13:14:07.976729 5534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:07.976994 5534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:07.977087 5534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:07.984041 5534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:07.984138 5534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:07.984180 5534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:07.984233 5534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:07.984274 5534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 13:14:07.984331 5534 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 13:14:07.984369 5534 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38561/jwks.json
I20260619 13:14:07.993696 5534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38561/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:07.994875 5534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:07.996078 5534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:08.006814 5540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:08.006778 5542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:08.006857 5539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:08.007670 5534 server_base.cc:1034] running on GCE node
I20260619 13:14:08.008442 5534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:08.009816 5534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:08.011068 5534 hybrid_clock.cc:648] HybridClock initialized: now 1781874848011010 us; error 54 us; skew 500 ppm
I20260619 13:14:08.011395 5534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:08.013499 5534 webserver.cc:460] Webserver started at http://127.0.97.62:33985/ using document root <none> and password file <none>
I20260619 13:14:08.014184 5534 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:08.014285 5534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:08.014567 5534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:08.017289 5534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "a217248a30ee4a40b7135217f86bc217"
format_stamp: "Formatted at 2026-06-19 13:14:08 on dist-test-slave-mkt3"
server_key: "265298665c0fba75be19d336f57b8504"
server_key_iv: "d939beb8dfba4a203a69d4f485016f53"
server_key_version: "encryptionkey@0"
I20260619 13:14:08.018105 5534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "a217248a30ee4a40b7135217f86bc217"
format_stamp: "Formatted at 2026-06-19 13:14:08 on dist-test-slave-mkt3"
server_key: "265298665c0fba75be19d336f57b8504"
server_key_iv: "d939beb8dfba4a203a69d4f485016f53"
server_key_version: "encryptionkey@0"
I20260619 13:14:08.022956 5534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:14:08.026265 5548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:08.027602 5534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:14:08.027760 5534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "a217248a30ee4a40b7135217f86bc217"
format_stamp: "Formatted at 2026-06-19 13:14:08 on dist-test-slave-mkt3"
server_key: "265298665c0fba75be19d336f57b8504"
server_key_iv: "d939beb8dfba4a203a69d4f485016f53"
server_key_version: "encryptionkey@0"
I20260619 13:14:08.027944 5534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:08.041702 5534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:08.042456 5534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:08.042693 5534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:08.058851 5534 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38475
I20260619 13:14:08.058871 5599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38475 every 8 connection(s)
I20260619 13:14:08.060072 5534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:14:08.060657 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 5534
I20260619 13:14:08.060846 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:14:08.061570 388 external_mini_cluster.cc:1428] Setting key 0c78b24c7625905f9433f91cdf51af2e
I20260619 13:14:08.064280 5600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:08.076402 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:08.063334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.0.97.62:38475)
0619 13:14:08.064322 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:08.064337 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:08.064384 (+ 47us) server_negotiation.cc:407] Connection header received
0619 13:14:08.066529 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:08.066555 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:08.067123 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:08.067544 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:08.068076 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:08.070090 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:08.071392 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:08.072323 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:08.072488 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:08.073732 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:08.073742 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:08.073759 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:08.074490 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:08.074580 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:08.074608 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:08.074725 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:08.074906 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 13:14:08.075235 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":285,"threads_started":1}
I20260619 13:14:08.077286 5600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217: Bootstrap starting.
I20260619 13:14:08.079800 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:14:08.080368 5600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:08.081269 5600 log.cc:824] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:08.083760 5600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217: No bootstrap required, opened a new log
I20260619 13:14:08.090433 5600 raft_consensus.cc:348] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a217248a30ee4a40b7135217f86bc217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38475 } }
I20260619 13:14:08.090754 5600 raft_consensus.cc:374] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:08.090843 5600 raft_consensus.cc:729] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a217248a30ee4a40b7135217f86bc217, State: Initialized, Role: FOLLOWER
I20260619 13:14:08.091341 5600 consensus_queue.cc:260] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a217248a30ee4a40b7135217f86bc217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38475 } }
I20260619 13:14:08.091562 5600 raft_consensus.cc:388] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:14:08.091684 5600 raft_consensus.cc:482] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:14:08.091866 5600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:08.093799 5600 raft_consensus.cc:504] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a217248a30ee4a40b7135217f86bc217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38475 } }
I20260619 13:14:08.094290 5600 leader_election.cc:302] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a217248a30ee4a40b7135217f86bc217; no voters:
I20260619 13:14:08.094779 5600 leader_election.cc:288] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:14:08.094902 5605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:08.095489 5605 raft_consensus.cc:686] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [term 1 LEADER]: Becoming Leader. State: Replica: a217248a30ee4a40b7135217f86bc217, State: Running, Role: LEADER
I20260619 13:14:08.095919 5605 consensus_queue.cc:237] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a217248a30ee4a40b7135217f86bc217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38475 } }
I20260619 13:14:08.096557 5600 sys_catalog.cc:564] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:08.098381 5606 sys_catalog.cc:455] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a217248a30ee4a40b7135217f86bc217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a217248a30ee4a40b7135217f86bc217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38475 } } }
I20260619 13:14:08.098719 5606 sys_catalog.cc:458] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:08.098824 5607 sys_catalog.cc:455] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a217248a30ee4a40b7135217f86bc217. Latest consensus state: current_term: 1 leader_uuid: "a217248a30ee4a40b7135217f86bc217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a217248a30ee4a40b7135217f86bc217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38475 } } }
I20260619 13:14:08.099114 5607 sys_catalog.cc:458] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:08.102147 5611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:14:08.106827 5611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:14:08.114727 5611 catalog_manager.cc:1347] Generated new cluster ID: 0a3ae9f8b22f4986b9b3266dbe953b5a
I20260619 13:14:08.114823 5611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:14:08.135120 5611 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:14:08.136803 5611 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:14:08.157711 5611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a217248a30ee4a40b7135217f86bc217: Generated new TSK 0
I20260619 13:14:08.158807 5611 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 13:14:08.197830 5630 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 13:14:08.198024 5630 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 13:14:08.199095 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:08.177621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.0.97.62:38475)
0619 13:14:08.177802 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:08.177808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:08.178326 (+ 518us) server_negotiation.cc:407] Connection header received
0619 13:14:08.178582 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:08.178587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:08.178694 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:08.178838 (+ 144us) server_negotiation.cc:226] Negotiated authn=JWT
0619 13:14:08.179273 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:08.180366 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:08.181708 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:08.182403 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:08.182516 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:08.182972 (+ 456us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 13:14:08.198438 (+ 15466us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 13:14:08.198797 (+ 359us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35390: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 13:14:08.199211 5603 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35390: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 13:14:08.200399 5629 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:08.177597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38475 (local address 127.0.0.1:35390)
0619 13:14:08.178231 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:08.178252 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:08.178390 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:08.178849 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:08.178853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:08.178862 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 13:14:08.179142 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:08.179149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:08.180475 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:08.180478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:08.181578 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:08.181589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:08.182508 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:08.182546 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:08.182666 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:08.182731 (+ 65us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 13:14:08.198737 (+ 16006us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 13:14:08.198990 (+ 253us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:38475: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":327,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:38475: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 13:14:08.202417 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 5534
I20260619 13:14:08.203472 5595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:08.294641 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 5534
2026-06-19T13:14:08Z chronyd exiting
I20260619 13:14:08.340981 388 test_util.cc:175] -----------------------------------------------
I20260619 13:14:08.341147 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781874807295266-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (703 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 13:14:08.344787 388 mini_oidc.cc:152] Starting JWKS server
I20260619 13:14:08.346573 388 webserver.cc:460] Webserver started at https://127.0.0.1:44111/ using document root <none> and password file <none>
I20260619 13:14:08.346769 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 13:14:08.347370 388 webserver.cc:460] Webserver started at http://127.0.0.1:33423/ using document root <none> and password file <none>
2026-06-19T13:14:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:14:08Z Disabled control of system clock
I20260619 13:14:08.370704 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36293
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36293
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44111/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/testchainca.cert with env {}
W20260619 13:14:08.674615 5637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:08.674878 5637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:08.674970 5637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:08.681934 5637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:08.682030 5637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:08.682072 5637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:08.682125 5637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:08.682168 5637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 13:14:08.682230 5637 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 13:14:08.682268 5637 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44111/jwks.json
I20260619 13:14:08.691675 5637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36293
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44111/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:08.692829 5637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:08.694053 5637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:08.704712 5645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:08.704756 5642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:08.704782 5643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:08.705391 5637 server_base.cc:1034] running on GCE node
I20260619 13:14:08.706099 5637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:08.707355 5637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:08.708534 5637 hybrid_clock.cc:648] HybridClock initialized: now 1781874848708478 us; error 54 us; skew 500 ppm
I20260619 13:14:08.708837 5637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:08.710927 5637 webserver.cc:460] Webserver started at http://127.0.97.62:38767/ using document root <none> and password file <none>
I20260619 13:14:08.711545 5637 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:08.711634 5637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:08.711925 5637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:08.714612 5637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "6715f404293147ba86591dc28ce3b850"
format_stamp: "Formatted at 2026-06-19 13:14:08 on dist-test-slave-mkt3"
server_key: "6b3bc69709c87c743d8d75562892eba5"
server_key_iv: "8964dc66bd0f29c68cc22dad51a00648"
server_key_version: "encryptionkey@0"
I20260619 13:14:08.715345 5637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "6715f404293147ba86591dc28ce3b850"
format_stamp: "Formatted at 2026-06-19 13:14:08 on dist-test-slave-mkt3"
server_key: "6b3bc69709c87c743d8d75562892eba5"
server_key_iv: "8964dc66bd0f29c68cc22dad51a00648"
server_key_version: "encryptionkey@0"
I20260619 13:14:08.720021 5637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:14:08.723229 5651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:08.724491 5637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:14:08.724640 5637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "6715f404293147ba86591dc28ce3b850"
format_stamp: "Formatted at 2026-06-19 13:14:08 on dist-test-slave-mkt3"
server_key: "6b3bc69709c87c743d8d75562892eba5"
server_key_iv: "8964dc66bd0f29c68cc22dad51a00648"
server_key_version: "encryptionkey@0"
I20260619 13:14:08.724822 5637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:08.748119 5637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:08.748854 5637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:08.749094 5637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:08.765616 5637 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36293
I20260619 13:14:08.765650 5702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36293 every 8 connection(s)
I20260619 13:14:08.766832 5637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:14:08.769635 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 5637
I20260619 13:14:08.769805 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:14:08.770349 5703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:08.770381 388 external_mini_cluster.cc:1428] Setting key 4111ecbd23e2565e17a75f7c02b8c18f
I20260619 13:14:08.784324 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:08.772055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.0.97.62:36293)
0619 13:14:08.773081 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:08.773099 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:08.773147 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:14:08.774886 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:08.774916 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:08.775486 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:08.775913 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:08.776406 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:08.778344 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:08.779530 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:08.780502 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:08.780683 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:08.781683 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:08.781693 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:08.781716 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:08.782438 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:08.782537 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:08.782571 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:08.782672 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:08.782852 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 13:14:08.783140 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":308,"threads_started":1}
I20260619 13:14:08.784576 5703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850: Bootstrap starting.
I20260619 13:14:08.787577 5703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:08.787817 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:14:08.788509 5703 log.cc:824] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:08.791095 5703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850: No bootstrap required, opened a new log
I20260619 13:14:08.798121 5703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6715f404293147ba86591dc28ce3b850" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36293 } }
I20260619 13:14:08.798437 5703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:08.798518 5703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6715f404293147ba86591dc28ce3b850, State: Initialized, Role: FOLLOWER
I20260619 13:14:08.799028 5703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6715f404293147ba86591dc28ce3b850" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36293 } }
I20260619 13:14:08.799263 5703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:14:08.799379 5703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:14:08.799518 5703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:08.801506 5703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6715f404293147ba86591dc28ce3b850" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36293 } }
I20260619 13:14:08.802004 5703 leader_election.cc:302] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6715f404293147ba86591dc28ce3b850; no voters:
I20260619 13:14:08.802584 5703 leader_election.cc:288] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:14:08.802711 5708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:08.803318 5708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [term 1 LEADER]: Becoming Leader. State: Replica: 6715f404293147ba86591dc28ce3b850, State: Running, Role: LEADER
I20260619 13:14:08.803782 5708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6715f404293147ba86591dc28ce3b850" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36293 } }
I20260619 13:14:08.804421 5703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:08.806262 5709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6715f404293147ba86591dc28ce3b850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6715f404293147ba86591dc28ce3b850" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36293 } } }
I20260619 13:14:08.806670 5709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:08.806746 5710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6715f404293147ba86591dc28ce3b850. Latest consensus state: current_term: 1 leader_uuid: "6715f404293147ba86591dc28ce3b850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6715f404293147ba86591dc28ce3b850" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36293 } } }
I20260619 13:14:08.807039 5710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:08.808490 5717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:14:08.813090 5717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:14:08.821004 5717 catalog_manager.cc:1347] Generated new cluster ID: f891c9d89d4c4cc9b28dd0df6652c175
I20260619 13:14:08.821121 5717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:14:08.852500 5717 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:14:08.854249 5717 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:14:08.867877 5717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6715f404293147ba86591dc28ce3b850: Generated new TSK 0
I20260619 13:14:08.868808 5717 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 13:14:08.913749 5733 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 13:14:08.913878 5733 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 13:14:08.915148 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:08.889885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.0.97.62:36293)
0619 13:14:08.890110 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:08.890117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:08.890537 (+ 420us) server_negotiation.cc:407] Connection header received
0619 13:14:08.890812 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:08.890816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:08.890890 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:08.891022 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0619 13:14:08.891543 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:08.892274 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:08.894250 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:08.894729 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:08.894830 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:08.895385 (+ 555us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 13:14:08.914457 (+ 19072us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 13:14:08.914841 (+ 384us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41184: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 13:14:08.915167 5732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:08.889794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36293 (local address 127.0.0.1:41184)
0619 13:14:08.890431 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:08.890450 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:08.890621 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:08.891060 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:08.891065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:08.891078 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0619 13:14:08.891395 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:08.891406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:08.892396 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:08.892402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:08.894102 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:08.894118 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:08.894845 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:08.894910 (+ 65us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:08.895103 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:08.895148 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 13:14:08.914683 (+ 19535us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 13:14:08.914872 (+ 189us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:36293: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":333,"threads_started":1}
W20260619 13:14:08.915275 5706 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41184: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 13:14:08.916312 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 5637
I20260619 13:14:08.917550 5698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:09.011018 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 5637
2026-06-19T13:14:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (718 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 13:14:09.063511 388 mini_oidc.cc:152] Starting JWKS server
I20260619 13:14:09.065204 388 webserver.cc:460] Webserver started at https://127.0.0.1:36599/ using document root <none> and password file <none>
I20260619 13:14:09.065405 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 13:14:09.065965 388 webserver.cc:460] Webserver started at http://127.0.0.1:46015/ using document root <none> and password file <none>
2026-06-19T13:14:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:14:09Z Disabled control of system clock
I20260619 13:14:09.088120 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40143
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40143
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36599/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 13:14:09.384878 5740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:09.385135 5740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:09.385251 5740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:09.392475 5740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:09.392577 5740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:09.392622 5740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:09.392697 5740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:09.392755 5740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 13:14:09.392836 5740 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 13:14:09.392882 5740 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36599/jwks.json
I20260619 13:14:09.401787 5740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36599/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:09.403095 5740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:09.404812 5740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:09.415344 5745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:09.415380 5746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:09.415289 5748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:09.416188 5740 server_base.cc:1034] running on GCE node
I20260619 13:14:09.416909 5740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:09.418217 5740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:09.419413 5740 hybrid_clock.cc:648] HybridClock initialized: now 1781874849419373 us; error 37 us; skew 500 ppm
I20260619 13:14:09.419720 5740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:09.426354 5740 webserver.cc:460] Webserver started at http://127.0.97.62:37141/ using document root <none> and password file <none>
I20260619 13:14:09.427012 5740 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:09.427106 5740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:09.427402 5740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:09.430212 5740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "b6c6c902faf6428da5f165b524ba5611"
format_stamp: "Formatted at 2026-06-19 13:14:09 on dist-test-slave-mkt3"
server_key: "f69009a8a8e59dcf220c941bde233583"
server_key_iv: "79c4ecb162c61ee2a4ccda5a0cda245b"
server_key_version: "encryptionkey@0"
I20260619 13:14:09.430948 5740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "b6c6c902faf6428da5f165b524ba5611"
format_stamp: "Formatted at 2026-06-19 13:14:09 on dist-test-slave-mkt3"
server_key: "f69009a8a8e59dcf220c941bde233583"
server_key_iv: "79c4ecb162c61ee2a4ccda5a0cda245b"
server_key_version: "encryptionkey@0"
I20260619 13:14:09.435804 5740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:14:09.438987 5754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:09.440184 5740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 13:14:09.440361 5740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "b6c6c902faf6428da5f165b524ba5611"
format_stamp: "Formatted at 2026-06-19 13:14:09 on dist-test-slave-mkt3"
server_key: "f69009a8a8e59dcf220c941bde233583"
server_key_iv: "79c4ecb162c61ee2a4ccda5a0cda245b"
server_key_version: "encryptionkey@0"
I20260619 13:14:09.440557 5740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:09.452600 5740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:09.453305 5740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:09.453534 5740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:09.469480 5740 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40143
I20260619 13:14:09.469501 5805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40143 every 8 connection(s)
I20260619 13:14:09.470726 5740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:14:09.473325 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 5740
I20260619 13:14:09.473495 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:14:09.474138 388 external_mini_cluster.cc:1428] Setting key dcba238282cfb7e50826be31f4091fa9
I20260619 13:14:09.474429 5806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:09.488364 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:09.475815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44624 (local address 127.0.97.62:40143)
0619 13:14:09.476804 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:09.476822 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:09.476869 (+ 47us) server_negotiation.cc:407] Connection header received
0619 13:14:09.478900 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:09.478926 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:09.479498 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:09.479938 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:09.480452 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:09.482402 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:09.483597 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:09.484569 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:09.484742 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:09.485763 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:09.485774 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:09.485791 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:09.486505 (+ 714us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:09.486604 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:09.486627 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:09.486733 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:09.486899 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 13:14:09.487192 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":308,"threads_started":1}
I20260619 13:14:09.488552 5806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611: Bootstrap starting.
I20260619 13:14:09.491604 5806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:09.491768 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:14:09.492545 5806 log.cc:824] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:09.495307 5806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611: No bootstrap required, opened a new log
I20260619 13:14:09.502394 5806 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c6c902faf6428da5f165b524ba5611" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40143 } }
I20260619 13:14:09.502701 5806 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:09.502786 5806 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c6c902faf6428da5f165b524ba5611, State: Initialized, Role: FOLLOWER
I20260619 13:14:09.503288 5806 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c6c902faf6428da5f165b524ba5611" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40143 } }
I20260619 13:14:09.503479 5806 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:14:09.503595 5806 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:14:09.503729 5806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:09.505671 5806 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c6c902faf6428da5f165b524ba5611" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40143 } }
I20260619 13:14:09.506165 5806 leader_election.cc:302] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6c6c902faf6428da5f165b524ba5611; no voters:
I20260619 13:14:09.506711 5806 leader_election.cc:288] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:14:09.506847 5811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:09.507730 5811 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [term 1 LEADER]: Becoming Leader. State: Replica: b6c6c902faf6428da5f165b524ba5611, State: Running, Role: LEADER
I20260619 13:14:09.508256 5811 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c6c902faf6428da5f165b524ba5611" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40143 } }
I20260619 13:14:09.508602 5806 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:09.510777 5812 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6c6c902faf6428da5f165b524ba5611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c6c902faf6428da5f165b524ba5611" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40143 } } }
I20260619 13:14:09.510855 5813 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6c6c902faf6428da5f165b524ba5611. Latest consensus state: current_term: 1 leader_uuid: "b6c6c902faf6428da5f165b524ba5611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6c6c902faf6428da5f165b524ba5611" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40143 } } }
I20260619 13:14:09.511207 5812 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:09.511216 5813 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:09.512722 5817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:14:09.517452 5817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:14:09.525800 5817 catalog_manager.cc:1347] Generated new cluster ID: f4c5c78722d14c0e965de828b5e773d3
I20260619 13:14:09.525928 5817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:14:09.557644 5817 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:14:09.558905 5817 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:14:09.568848 5817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6c6c902faf6428da5f165b524ba5611: Generated new TSK 0
I20260619 13:14:09.569588 5817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:14:09.596546 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:09.588567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.0.97.62:40143)
0619 13:14:09.588764 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:09.588772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:09.589176 (+ 404us) server_negotiation.cc:407] Connection header received
0619 13:14:09.589372 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:09.589376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:09.589444 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:09.589563 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0619 13:14:09.590081 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:09.590769 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:09.592756 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:09.593191 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:09.593281 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:09.593899 (+ 618us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 13:14:09.595952 (+ 2053us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36599/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 13:14:09.596267 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44630: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36599/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 13:14:09.596660 5809 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44630: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36599/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 13:14:09.598687 5835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:09.588458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40143 (local address 127.0.0.1:44630)
0619 13:14:09.589044 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:09.589079 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:09.589222 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:09.589591 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:09.589596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:09.589609 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0619 13:14:09.589935 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:09.589947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:09.590907 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:09.590913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:09.592613 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:09.592628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:09.593302 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:09.593375 (+ 73us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:09.593551 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:09.593592 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 13:14:09.596149 (+ 2557us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36599/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 13:14:09.598278 (+ 2129us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:40143: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36599/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":1858,"thread_start_us":310,"threads_started":1}
I20260619 13:14:09.600536 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 5740
I20260619 13:14:09.601532 5801 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 13:14:09.611451 5836 webserver.cc:533] Webserver: sslize failed:
E20260619 13:14:09.611563 5836 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 13:14:09.693802 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 5740
2026-06-19T13:14:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (681 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T13:14:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:14:09Z Disabled control of system clock
I20260619 13:14:09.766702 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43535
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43535
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/testchainca.cert with env {}
W20260619 13:14:10.067745 5841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:10.068020 5841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:10.068120 5841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:10.075040 5841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:10.075134 5841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:10.075184 5841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:10.075239 5841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:10.075281 5841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 13:14:10.075337 5841 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 13:14:10.075376 5841 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 13:14:10.084724 5841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:10.085947 5841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:10.087324 5841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:10.097975 5846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:10.098119 5849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:10.098169 5847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:10.098552 5841 server_base.cc:1034] running on GCE node
I20260619 13:14:10.099272 5841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:10.100533 5841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:10.101716 5841 hybrid_clock.cc:648] HybridClock initialized: now 1781874850101664 us; error 48 us; skew 500 ppm
I20260619 13:14:10.102034 5841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:10.104092 5841 webserver.cc:460] Webserver started at http://127.0.97.62:33767/ using document root <none> and password file <none>
I20260619 13:14:10.104727 5841 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:10.104817 5841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:10.105095 5841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:10.107829 5841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "3d56e32880d04d81aaa9b6e8673f8330"
format_stamp: "Formatted at 2026-06-19 13:14:10 on dist-test-slave-mkt3"
server_key: "0e499653c1870bcd8c2c88f0e253b074"
server_key_iv: "c6a4c85e38c1a2d8d5ff928d4bd45b9d"
server_key_version: "encryptionkey@0"
I20260619 13:14:10.108562 5841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "3d56e32880d04d81aaa9b6e8673f8330"
format_stamp: "Formatted at 2026-06-19 13:14:10 on dist-test-slave-mkt3"
server_key: "0e499653c1870bcd8c2c88f0e253b074"
server_key_iv: "c6a4c85e38c1a2d8d5ff928d4bd45b9d"
server_key_version: "encryptionkey@0"
I20260619 13:14:10.113283 5841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 13:14:10.116442 5855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:10.117791 5841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:14:10.117990 5841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "3d56e32880d04d81aaa9b6e8673f8330"
format_stamp: "Formatted at 2026-06-19 13:14:10 on dist-test-slave-mkt3"
server_key: "0e499653c1870bcd8c2c88f0e253b074"
server_key_iv: "c6a4c85e38c1a2d8d5ff928d4bd45b9d"
server_key_version: "encryptionkey@0"
I20260619 13:14:10.118183 5841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:10.135361 5841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:10.136129 5841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:10.136371 5841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:10.152657 5841 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43535
I20260619 13:14:10.152675 5906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43535 every 8 connection(s)
I20260619 13:14:10.153954 5841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:14:10.157579 5907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:10.161396 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 5841
I20260619 13:14:10.161549 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:14:10.162220 388 external_mini_cluster.cc:1428] Setting key 2463bc79ebad21e7a606a2dac8799a5e
I20260619 13:14:10.167989 5907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330: Bootstrap starting.
I20260619 13:14:10.171303 5907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:10.172463 5907 log.cc:824] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:10.175261 5907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330: No bootstrap required, opened a new log
I20260619 13:14:10.175357 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:10.163983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.0.97.62:43535)
0619 13:14:10.164857 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:10.164867 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:10.164907 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:14:10.166620 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:10.166642 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:10.167075 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:10.167381 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:10.167947 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:10.169238 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:10.171000 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:10.171801 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:10.171926 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:10.173064 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:10.173074 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:10.173094 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:10.173767 (+ 673us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:10.173867 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:10.173897 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:10.173995 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:10.174188 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 13:14:10.174492 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":284,"threads_started":1}
I20260619 13:14:10.178689 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:14:10.182554 5907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d56e32880d04d81aaa9b6e8673f8330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43535 } }
I20260619 13:14:10.182874 5907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:10.182956 5907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d56e32880d04d81aaa9b6e8673f8330, State: Initialized, Role: FOLLOWER
I20260619 13:14:10.183475 5907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d56e32880d04d81aaa9b6e8673f8330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43535 } }
I20260619 13:14:10.183681 5907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:14:10.183800 5907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:14:10.183946 5907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:10.186051 5907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d56e32880d04d81aaa9b6e8673f8330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43535 } }
I20260619 13:14:10.186491 5907 leader_election.cc:302] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d56e32880d04d81aaa9b6e8673f8330; no voters:
I20260619 13:14:10.186976 5907 leader_election.cc:288] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:14:10.187129 5912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:10.187728 5912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [term 1 LEADER]: Becoming Leader. State: Replica: 3d56e32880d04d81aaa9b6e8673f8330, State: Running, Role: LEADER
I20260619 13:14:10.188212 5912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d56e32880d04d81aaa9b6e8673f8330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43535 } }
I20260619 13:14:10.188719 5907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:10.190994 5913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d56e32880d04d81aaa9b6e8673f8330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d56e32880d04d81aaa9b6e8673f8330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43535 } } }
I20260619 13:14:10.191017 5914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d56e32880d04d81aaa9b6e8673f8330. Latest consensus state: current_term: 1 leader_uuid: "3d56e32880d04d81aaa9b6e8673f8330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d56e32880d04d81aaa9b6e8673f8330" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43535 } } }
I20260619 13:14:10.191391 5914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:10.191393 5913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:10.192269 5919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:14:10.201423 5919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:14:10.211262 5919 catalog_manager.cc:1347] Generated new cluster ID: c3f29f3fe27a490dbb23731646eb8b8b
I20260619 13:14:10.211361 5919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:14:10.220991 5919 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:14:10.222738 5919 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:14:10.235419 5919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d56e32880d04d81aaa9b6e8673f8330: Generated new TSK 0
I20260619 13:14:10.236466 5919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:14:10.299969 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:10.261447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.0.97.62:43535)
0619 13:14:10.261629 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:10.261635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:10.262041 (+ 406us) server_negotiation.cc:407] Connection header received
0619 13:14:10.262238 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:10.262244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:10.262328 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:10.262480 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0619 13:14:10.262850 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:10.263964 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:10.265266 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:10.265957 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:10.266093 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:10.266533 (+ 440us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 13:14:10.299256 (+ 32723us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 13:14:10.299686 (+ 430us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43194: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 13:14:10.300030 5936 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:10.261306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43535 (local address 127.0.0.1:43194)
0619 13:14:10.261959 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:10.261975 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:10.262088 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:10.262482 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:10.262485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:10.262494 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 13:14:10.262721 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:10.262728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:10.264079 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:10.264082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:10.265136 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:10.265147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:10.266076 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:10.266129 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:10.266248 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:10.266293 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 13:14:10.299506 (+ 33213us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 13:14:10.299749 (+ 243us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:43535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":353,"threads_started":1}
W20260619 13:14:10.300122 5910 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43194: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260619 13:14:10.301698 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 5841
I20260619 13:14:10.302793 5902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:10.394153 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 5841
2026-06-19T13:14:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (680 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (378 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (386 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:14:11 dist-test-slave-mkt3 krb5kdc[5951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:14:11 dist-test-slave-mkt3 krb5kdc[5951](info): set up 2 sockets
Jun 19 13:14:11 dist-test-slave-mkt3 krb5kdc[5951](info): commencing operation
krb5kdc: starting...
W20260619 13:14:12.146173 388 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:14:12 dist-test-slave-mkt3 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:14:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:14:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:14:12.328819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35055
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:12.630681 5964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:12.631008 5964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:12.631151 5964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:12.638062 5964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:12.638177 5964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:12.638252 5964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:12.638331 5964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:12.638391 5964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:14:12.647856 5964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:12.649091 5964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:12.650337 5964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:12.660800 5969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:12.660848 5970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:12.660872 5972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:12.661573 5964 server_base.cc:1034] running on GCE node
I20260619 13:14:12.662292 5964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:12.663504 5964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:12.664703 5964 hybrid_clock.cc:648] HybridClock initialized: now 1781874852664653 us; error 35 us; skew 500 ppm
I20260619 13:14:12.664999 5964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:12 dist-test-slave-mkt3 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:12.667739 5964 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:14:12.668952 5964 webserver.cc:460] Webserver started at http://127.0.97.62:36575/ using document root <none> and password file <none>
I20260619 13:14:12.669528 5964 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:12.669620 5964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:12.669885 5964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:12.672590 5964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "1aa4ec74790f4bcb85414e76db8e1f69"
format_stamp: "Formatted at 2026-06-19 13:14:12 on dist-test-slave-mkt3"
server_key: "a0fa2fd1e8dad3df5f0c362aac5ca896"
server_key_iv: "5ea0e7d8a3443bd99107ba0b3441f7f8"
server_key_version: "encryptionkey@0"
I20260619 13:14:12.673267 5964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "1aa4ec74790f4bcb85414e76db8e1f69"
format_stamp: "Formatted at 2026-06-19 13:14:12 on dist-test-slave-mkt3"
server_key: "a0fa2fd1e8dad3df5f0c362aac5ca896"
server_key_iv: "5ea0e7d8a3443bd99107ba0b3441f7f8"
server_key_version: "encryptionkey@0"
I20260619 13:14:12.677812 5964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 13:14:12.680866 5979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:12.682108 5964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 13:14:12.682233 5964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "1aa4ec74790f4bcb85414e76db8e1f69"
format_stamp: "Formatted at 2026-06-19 13:14:12 on dist-test-slave-mkt3"
server_key: "a0fa2fd1e8dad3df5f0c362aac5ca896"
server_key_iv: "5ea0e7d8a3443bd99107ba0b3441f7f8"
server_key_version: "encryptionkey@0"
I20260619 13:14:12.682381 5964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:12.699354 5964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:12.703140 5964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:12.703356 5964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:12.718990 5964 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35055
I20260619 13:14:12.719029 6030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35055 every 8 connection(s)
I20260619 13:14:12.720191 5964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:14:12.723551 6031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:12.725137 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 5964
I20260619 13:14:12.725306 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:14:12.725934 388 external_mini_cluster.cc:1428] Setting key 8ad005fbc2f0f9f575261c00867682bc
I20260619 13:14:12.735321 6031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69: Bootstrap starting.
I20260619 13:14:12.738600 6031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:12.739481 6031 log.cc:824] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:14:12 dist-test-slave-mkt3 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:12.742429 6031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69: No bootstrap required, opened a new log
I20260619 13:14:12.748615 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:12.727713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.0.97.62:35055)
0619 13:14:12.728467 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:12.728486 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:12.728534 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:14:12.730755 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:12.730782 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:12.731256 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:12.731675 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:12.732785 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:12.734669 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:12.735938 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:12.736918 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:12.737091 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:12.740957 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:12.740967 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:12.740985 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:12.741039 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:12.744608 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:12.745193 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:12.745205 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:12.745213 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:12.745414 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:12.745731 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:12.745736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:12.745739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:12.746407 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:12.746682 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:12.746953 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 13:14:12.747352 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":295,"threads_started":1}
I20260619 13:14:12.751950 6031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa4ec74790f4bcb85414e76db8e1f69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35055 } }
I20260619 13:14:12.752249 6031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:12.752317 6031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aa4ec74790f4bcb85414e76db8e1f69, State: Initialized, Role: FOLLOWER
I20260619 13:14:12.752803 6031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa4ec74790f4bcb85414e76db8e1f69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35055 } }
I20260619 13:14:12.752988 6031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:14:12.753079 6031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:14:12.753190 6031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:12.755134 6031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa4ec74790f4bcb85414e76db8e1f69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35055 } }
I20260619 13:14:12.755543 6031 leader_election.cc:302] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aa4ec74790f4bcb85414e76db8e1f69; no voters:
I20260619 13:14:12.756017 6031 leader_election.cc:288] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:14:12.756139 6036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:12.756711 6036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [term 1 LEADER]: Becoming Leader. State: Replica: 1aa4ec74790f4bcb85414e76db8e1f69, State: Running, Role: LEADER
I20260619 13:14:12.757192 6036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa4ec74790f4bcb85414e76db8e1f69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35055 } }
I20260619 13:14:12.757975 6031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:12.759862 6038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1aa4ec74790f4bcb85414e76db8e1f69. Latest consensus state: current_term: 1 leader_uuid: "1aa4ec74790f4bcb85414e76db8e1f69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa4ec74790f4bcb85414e76db8e1f69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35055 } } }
I20260619 13:14:12.759955 6037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1aa4ec74790f4bcb85414e76db8e1f69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aa4ec74790f4bcb85414e76db8e1f69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35055 } } }
I20260619 13:14:12.760762 6037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:12.761746 6038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:12.762976 6044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:14:12.767060 6044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:14:12.774961 6044 catalog_manager.cc:1347] Generated new cluster ID: 3a1b8d7416ec40fd8605b53c489d3370
I20260619 13:14:12.775090 6044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:14:12.792337 6044 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:14:12.794195 6044 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:14:12.808387 6044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1aa4ec74790f4bcb85414e76db8e1f69: Generated new TSK 0
I20260619 13:14:12.809448 6044 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:14:12.850574 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35055
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:13.162608 6057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:13.162899 6057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:13.162999 6057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:13.170162 6057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:13.170272 6057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:13.170393 6057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:14:13.179862 6057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:13.181118 6057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:13.182375 6057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:13.194607 6063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:13.194568 6065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:13.194628 6062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:13.195330 6057 server_base.cc:1034] running on GCE node
I20260619 13:14:13.195863 6057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:13.196717 6057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:13.197922 6057 hybrid_clock.cc:648] HybridClock initialized: now 1781874853197878 us; error 35 us; skew 500 ppm
I20260619 13:14:13.198202 6057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:13 dist-test-slave-mkt3 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:13.201049 6057 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:14:13.202401 6057 webserver.cc:460] Webserver started at http://127.0.97.1:34871/ using document root <none> and password file <none>
I20260619 13:14:13.203066 6057 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:13.203162 6057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:13.203408 6057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:13.206182 6057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "8656dcaa2c2a4226abff5189d9a652e9"
format_stamp: "Formatted at 2026-06-19 13:14:13 on dist-test-slave-mkt3"
server_key: "9ec9a42bc526902a0c33005839343102"
server_key_iv: "a1dbcea4e324f05a5aa08844df5b7586"
server_key_version: "encryptionkey@0"
I20260619 13:14:13.206904 6057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8656dcaa2c2a4226abff5189d9a652e9"
format_stamp: "Formatted at 2026-06-19 13:14:13 on dist-test-slave-mkt3"
server_key: "9ec9a42bc526902a0c33005839343102"
server_key_iv: "a1dbcea4e324f05a5aa08844df5b7586"
server_key_version: "encryptionkey@0"
I20260619 13:14:13.211567 6057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:14:13.214671 6072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:13.215901 6057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:14:13.216038 6057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "8656dcaa2c2a4226abff5189d9a652e9"
format_stamp: "Formatted at 2026-06-19 13:14:13 on dist-test-slave-mkt3"
server_key: "9ec9a42bc526902a0c33005839343102"
server_key_iv: "a1dbcea4e324f05a5aa08844df5b7586"
server_key_version: "encryptionkey@0"
I20260619 13:14:13.216218 6057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:13.232805 6057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:13.236820 6057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:13.237079 6057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:13.238010 6057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:13.239229 6057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:13.239307 6057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:13.239420 6057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:13.239475 6057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:13.267503 6057 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39631
I20260619 13:14:13.267549 6184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39631 every 8 connection(s)
I20260619 13:14:13.268723 6057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:14:13.277508 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 6057
I20260619 13:14:13.277697 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:14:13.278394 388 external_mini_cluster.cc:1428] Setting key b4e38e01ef0cba0026192a72131e1b28
Jun 19 13:14:13 dist-test-slave-mkt3 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:13.287556 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:13.270981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53903 (local address 127.0.97.62:35055)
0619 13:14:13.271369 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:13.271373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:13.272351 (+ 978us) server_negotiation.cc:407] Connection header received
0619 13:14:13.274029 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:13.274034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:13.274105 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:13.274195 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:13.275569 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:13.276281 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:13.278091 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:13.278488 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:13.278624 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:13.282256 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:13.282262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:13.282266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:13.282294 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:13.284606 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:13.285424 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:13.285429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:13.285433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:13.285574 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:13.285981 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:13.285986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:13.285989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:13.286318 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:13.286440 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:13.287159 (+ 719us) server_negotiation.cc:299] Negotiation successful
0619 13:14:13.287314 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":196,"threads_started":1}
I20260619 13:14:13.288813 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:13.271539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35055 (local address 127.0.97.1:53903)
0619 13:14:13.272201 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:13.272243 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:13.273753 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:13.274425 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:13.274433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:13.274893 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:13.275414 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:13.275432 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:13.276419 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:13.276423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:13.277956 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:13.277971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:13.278708 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:13.278716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:13.278997 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:13.279626 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:13.279658 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:13.282055 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:13.284744 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:13.284752 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:13.284767 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:13.285304 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:13.285670 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:13.285673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:13.285675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:13.285872 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:13.286436 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:13.286444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:13.286822 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 13:14:13.287088 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 13:14:13.287580 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":264,"threads_started":1}
I20260619 13:14:13.290220 6185 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35055
I20260619 13:14:13.290530 6185 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:13.291317 6185 heartbeater.cc:511] Master 127.0.97.62:35055 requested a full tablet report, sending...
I20260619 13:14:13.293370 5996 ts_manager.cc:194] Registered new tserver with Master: 8656dcaa2c2a4226abff5189d9a652e9 (127.0.97.1:39631)
I20260619 13:14:13.295009 5996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53903
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:14:13.317411 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35055
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:13.614765 6191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:13.615051 6191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:13.615186 6191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:13.622362 6191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:13.622489 6191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:13.622643 6191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:14:13.632030 6191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:13.633307 6191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:13.634588 6191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:13.646299 6196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:13.646330 6197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:13.646359 6199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:13.647151 6191 server_base.cc:1034] running on GCE node
I20260619 13:14:13.647660 6191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:13.648453 6191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:13.649629 6191 hybrid_clock.cc:648] HybridClock initialized: now 1781874853649588 us; error 36 us; skew 500 ppm
I20260619 13:14:13.649952 6191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:13 dist-test-slave-mkt3 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:13.652829 6191 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:14:13.654322 6191 webserver.cc:460] Webserver started at http://127.0.97.2:39479/ using document root <none> and password file <none>
I20260619 13:14:13.654903 6191 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:13.654996 6191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:13.655220 6191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:13.658008 6191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "e88012b8d5b64703bc10c4a2841556d5"
format_stamp: "Formatted at 2026-06-19 13:14:13 on dist-test-slave-mkt3"
server_key: "0e68c30806ed124321b17afee420c8b2"
server_key_iv: "1ec23e4bba71bbf70583c4e2dac72f27"
server_key_version: "encryptionkey@0"
I20260619 13:14:13.658733 6191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e88012b8d5b64703bc10c4a2841556d5"
format_stamp: "Formatted at 2026-06-19 13:14:13 on dist-test-slave-mkt3"
server_key: "0e68c30806ed124321b17afee420c8b2"
server_key_iv: "1ec23e4bba71bbf70583c4e2dac72f27"
server_key_version: "encryptionkey@0"
I20260619 13:14:13.663434 6191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:14:13.666555 6206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:13.667769 6191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 13:14:13.667900 6191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "e88012b8d5b64703bc10c4a2841556d5"
format_stamp: "Formatted at 2026-06-19 13:14:13 on dist-test-slave-mkt3"
server_key: "0e68c30806ed124321b17afee420c8b2"
server_key_iv: "1ec23e4bba71bbf70583c4e2dac72f27"
server_key_version: "encryptionkey@0"
I20260619 13:14:13.668064 6191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:13.687799 6191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:13.691798 6191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:13.692054 6191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:13.692941 6191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:13.694203 6191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:13.694283 6191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:13.694402 6191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:13.694451 6191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:13.726413 6191 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35423
I20260619 13:14:13.726423 6318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35423 every 8 connection(s)
I20260619 13:14:13.727659 6191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:14:13.731854 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 6191
I20260619 13:14:13.732000 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:14:13.732417 388 external_mini_cluster.cc:1428] Setting key 2442e9222cc738690b9b50d4ce0ae298
Jun 19 13:14:13 dist-test-slave-mkt3 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:13.747967 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:13.730078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55111 (local address 127.0.97.62:35055)
0619 13:14:13.730268 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:13.730274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:13.731516 (+ 1242us) server_negotiation.cc:407] Connection header received
0619 13:14:13.733939 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:13.733944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:13.734013 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:13.734094 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:13.735635 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:13.736633 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:13.738163 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:13.738774 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:13.738876 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:13.742803 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:13.742807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:13.742811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:13.742840 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:13.745165 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:13.745931 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:13.745935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:13.745937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:13.746055 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:13.746470 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:13.746472 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:13.746474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:13.746707 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:13.746806 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:13.747590 (+ 784us) server_negotiation.cc:299] Negotiation successful
0619 13:14:13.747747 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:14:13.749394 6321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:13.730581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35055 (local address 127.0.97.2:55111)
0619 13:14:13.731363 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:13.731408 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:13.733644 (+ 2236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:13.734346 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:13.734358 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:13.734922 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:13.735463 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:13.735480 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:13.736758 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:13.736761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:13.738008 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:13.738019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:13.739179 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:13.739184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:13.739379 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:13.740031 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:13.740064 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:13.742596 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:13.745285 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:13.745293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:13.745309 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:13.745821 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:13.746149 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:13.746154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:13.746157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:13.746380 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:13.746812 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:13.746820 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:13.747224 (+ 404us) client_negotiation.cc:769] Sending connection context
0619 13:14:13.747515 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 13:14:13.748043 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":288,"threads_started":1}
I20260619 13:14:13.750717 6319 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35055
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:14:13.751029 6319 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:13.751742 6319 heartbeater.cc:511] Master 127.0.97.62:35055 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 13:14:13.753108 5996 ts_manager.cc:194] Registered new tserver with Master: e88012b8d5b64703bc10c4a2841556d5 (127.0.97.2:35423)
I20260619 13:14:13.753873 5996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:14:13.772789 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35055
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:14.075392 6324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:14.075651 6324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:14.075738 6324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:14.082743 6324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:14.082856 6324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:14.082964 6324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:14:14.092446 6324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:14.093623 6324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:14.094878 6324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:14.107028 6329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:14.107000 6332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:14.107160 6330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:14.107923 6324 server_base.cc:1034] running on GCE node
I20260619 13:14:14.108511 6324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:14.109387 6324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:14.110591 6324 hybrid_clock.cc:648] HybridClock initialized: now 1781874854110546 us; error 45 us; skew 500 ppm
I20260619 13:14:14.110885 6324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:14.113809 6324 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:14:14.115172 6324 webserver.cc:460] Webserver started at http://127.0.97.3:41887/ using document root <none> and password file <none>
I20260619 13:14:14.115810 6324 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:14.115924 6324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:14.116185 6324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:14.118952 6324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "dba602847b064475ba96454b3969fe01"
format_stamp: "Formatted at 2026-06-19 13:14:14 on dist-test-slave-mkt3"
server_key: "212ad90220c70708d51b1841c90967f8"
server_key_iv: "3cbeef263f1a1769844ed02c97dbe341"
server_key_version: "encryptionkey@0"
I20260619 13:14:14.119717 6324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dba602847b064475ba96454b3969fe01"
format_stamp: "Formatted at 2026-06-19 13:14:14 on dist-test-slave-mkt3"
server_key: "212ad90220c70708d51b1841c90967f8"
server_key_iv: "3cbeef263f1a1769844ed02c97dbe341"
server_key_version: "encryptionkey@0"
I20260619 13:14:14.124444 6324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 13:14:14.127619 6339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:14.128849 6324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:14:14.128983 6324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "dba602847b064475ba96454b3969fe01"
format_stamp: "Formatted at 2026-06-19 13:14:14 on dist-test-slave-mkt3"
server_key: "212ad90220c70708d51b1841c90967f8"
server_key_iv: "3cbeef263f1a1769844ed02c97dbe341"
server_key_version: "encryptionkey@0"
I20260619 13:14:14.129141 6324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:14.146525 6324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:14.150283 6324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:14.150527 6324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:14.151408 6324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:14.152661 6324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:14.152745 6324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:14.152858 6324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:14.152909 6324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:14.180649 6324 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36543
I20260619 13:14:14.180692 6451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36543 every 8 connection(s)
I20260619 13:14:14.181821 6324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:14:14.188623 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 6324
I20260619 13:14:14.188791 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:14:14.189206 388 external_mini_cluster.cc:1428] Setting key 0b00f3280aed2d22ff31326be3234dd2
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:14.199893 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:14.184095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37041 (local address 127.0.97.62:35055)
0619 13:14:14.184265 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:14.184270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:14.185382 (+ 1112us) server_negotiation.cc:407] Connection header received
0619 13:14:14.187132 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:14.187136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:14.187198 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:14.187294 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:14.188878 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.189613 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.191525 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.191977 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.192076 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:14.195130 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:14.195134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:14.195137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:14.195168 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:14.197505 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:14.198172 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:14.198176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:14.198179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:14.198304 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:14.198618 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:14.198621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:14.198623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:14.198865 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:14.198966 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:14.199519 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 13:14:14.199659 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:14:14.200765 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:14.184532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35055 (local address 127.0.97.3:37041)
0619 13:14:14.185202 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:14.185265 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:14.186906 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:14.187479 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:14.187487 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:14.187975 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:14.188675 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:14.188703 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.189761 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.189766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:14.191402 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:14.191416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.192181 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.192188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:14.192398 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:14.192930 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:14.192954 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:14.194946 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:14.197618 (+ 2672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:14.197623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:14.197635 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:14.198059 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:14.198383 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:14.198385 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:14.198387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:14.198543 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:14.198949 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:14.198955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:14.199221 (+ 266us) client_negotiation.cc:769] Sending connection context
0619 13:14:14.199447 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 13:14:14.199786 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":264,"threads_started":1}
I20260619 13:14:14.201975 6452 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35055
I20260619 13:14:14.202257 6452 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:14.202972 6452 heartbeater.cc:511] Master 127.0.97.62:35055 requested a full tablet report, sending...
I20260619 13:14:14.204250 5996 ts_manager.cc:194] Registered new tserver with Master: dba602847b064475ba96454b3969fe01 (127.0.97.3:36543)
I20260619 13:14:14.204996 5996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37041
I20260619 13:14:14.215929 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/test-admin.keytab.
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:14.238615 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 13:14:14.243149 6459 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:14.240906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45540 (local address 127.0.97.1:39631)
0619 13:14:14.241455 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:14.241465 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:14.241497 (+ 32us) server_negotiation.cc:407] Connection header received
0619 13:14:14.241637 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:14.241645 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:14.241932 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:14.242093 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:14.242824 (+ 731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45540: BlockingRecv error: recv got EOF from 127.0.0.1:45540 (error 108)
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":263,"threads_started":1}
W20260619 13:14:14.243466 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:14.245105 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:14.246337 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:14:14.259855 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:14.247691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.0.97.1:39631)
0619 13:14:14.247867 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:14.247872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:14.248094 (+ 222us) server_negotiation.cc:407] Connection header received
0619 13:14:14.248243 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:14.248246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:14.248300 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:14.248367 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:14.249196 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.250310 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.251425 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.251939 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.252025 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:14.255107 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:14.255117 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:14.255130 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:14.255167 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:14.257443 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:14.258061 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:14.258067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:14.258069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:14.258179 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:14.258484 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:14.258486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:14.258488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:14.258783 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:14.259076 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:14.259390 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 13:14:14.259617 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 13:14:14.264853 6459 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:14.263429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45556 (local address 127.0.97.1:39631)
0619 13:14:14.263570 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:14.263575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:14.263892 (+ 317us) server_negotiation.cc:407] Connection header received
0619 13:14:14.264016 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:14.264019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:14.264076 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:14.264140 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:14.264656 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45556: BlockingRecv error: recv got EOF from 127.0.0.1:45556 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 13:14:14.265066 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:14.266427 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:14.267839 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:14:14.281936 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:14.269174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.0.97.1:39631)
0619 13:14:14.269361 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:14.269366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:14.269603 (+ 237us) server_negotiation.cc:407] Connection header received
0619 13:14:14.269735 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:14.269738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:14.269796 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:14.269906 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:14.270926 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.271646 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.273329 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.273775 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.273919 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:14.277563 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:14.277568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:14.277571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:14.277601 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:14.279925 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:14.280469 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:14.280473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:14.280475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:14.280577 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:14.280829 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:14.280833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:14.280836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:14.281081 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:14.281274 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:14.281463 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 13:14:14.281620 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 13:14:14.285775 6459 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:14.284107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.0.97.1:39631)
0619 13:14:14.284354 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:14.284359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:14.284542 (+ 183us) server_negotiation.cc:407] Connection header received
0619 13:14:14.284707 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:14.284713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:14.284786 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:14.284869 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:14.285563 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45586: BlockingRecv error: recv got EOF from 127.0.0.1:45586 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
W20260619 13:14:14.285950 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781874807295266-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:14.287353 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[5951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:14.288499 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:14:14.298022 6185 heartbeater.cc:503] Master 127.0.97.62:35055 was elected leader, sending a full tablet report...
I20260619 13:14:14.300989 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:14.289558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.0.97.1:39631)
0619 13:14:14.289747 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:14.289752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:14.289970 (+ 218us) server_negotiation.cc:407] Connection header received
0619 13:14:14.290087 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:14.290090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:14.290137 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:14.290200 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:14.291075 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.291761 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.292926 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.293326 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.293426 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:14.296507 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:14.296512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:14.296515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:14.296543 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:14.298936 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:14.299484 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:14.299488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:14.299491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:14.299627 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:14.299902 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:14.299905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:14.299907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:14.300136 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:14.300342 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:14.300530 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 13:14:14.300679 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:14:14.302387 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 6057
I20260619 13:14:14.312412 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:14.302949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.0.97.1:39631)
0619 13:14:14.303121 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:14.303126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:14.303358 (+ 232us) server_negotiation.cc:407] Connection header received
0619 13:14:14.303462 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:14.303465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:14.303513 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:14.303582 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:14.304444 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.305072 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.306269 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.306638 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.306728 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:14.308403 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:14.308408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:14.308411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:14.308436 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:14.310512 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:14.311033 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:14.311036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:14.311038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:14.311145 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:14.311404 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:14.311407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:14.311408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:14.311606 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:14.311766 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:14.311988 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 13:14:14.312115 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:14:14.312943 6180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:14.393019 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 6057
I20260619 13:14:14.405530 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 6191
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:14:14.419381 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:14.406320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.0.97.2:35423)
0619 13:14:14.406838 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:14.406845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:14.406871 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:14:14.406937 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:14.406944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:14.407157 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:14.407290 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:14.408112 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.409130 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.410389 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.410932 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.411023 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:14.414420 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:14.414431 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:14.414445 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:14.414480 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:14.416866 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:14.417472 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:14.417478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:14.417481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:14.417605 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:14.417972 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:14.417975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:14.417977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:14.418288 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:14.418582 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:14.418876 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 13:14:14.419106 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
I20260619 13:14:14.420603 6314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:14.499109 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 6191
I20260619 13:14:14.511296 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 6324
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[5951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:14:14.528836 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:14.512260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.0.97.3:36543)
0619 13:14:14.512857 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:14.512864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:14.512892 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:14:14.512954 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:14.512961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:14.513155 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:14.513291 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:14.514184 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.515158 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.516268 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:14.516786 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:14.516882 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:14.524199 (+ 7317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:14.524210 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:14.524224 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:14.524271 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:14.526667 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:14.527207 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:14.527212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:14.527215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:14.527336 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:14.527593 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:14.527595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:14.527597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:14.527905 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:14.528116 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:14.528327 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 13:14:14.528536 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":316,"threads_started":1}
I20260619 13:14:14.530061 6447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:14.608597 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 6324
I20260619 13:14:14.620538 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 5964
I20260619 13:14:14.621621 6026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:14.719295 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 5964
2026-06-19T13:14:14Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3563 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[6479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[6479](info): set up 2 sockets
Jun 19 13:14:14 dist-test-slave-mkt3 krb5kdc[6479](info): commencing operation
krb5kdc: starting...
W20260619 13:14:15.714184 388 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:14:15 dist-test-slave-mkt3 krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:14:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:14:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260619 13:14:15.890236 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45357
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:16.196244 6492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:16.196504 6492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:16.196589 6492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:16.203509 6492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:16.203615 6492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:16.203660 6492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:16.203716 6492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:16.203758 6492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:14:16.213183 6492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:16.214409 6492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:16.215641 6492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:16.226377 6500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:16.226423 6498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:16.226426 6497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:16.227120 6492 server_base.cc:1034] running on GCE node
I20260619 13:14:16.227873 6492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:16.229255 6492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:16.230446 6492 hybrid_clock.cc:648] HybridClock initialized: now 1781874856230389 us; error 45 us; skew 500 ppm
I20260619 13:14:16.230756 6492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:16 dist-test-slave-mkt3 krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:16.233600 6492 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260619 13:14:16.235026 6492 webserver.cc:460] Webserver started at http://127.0.97.62:34161/ using document root <none> and password file <none>
I20260619 13:14:16.235664 6492 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:16.235769 6492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:16.236042 6492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:16.239081 6492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe"
format_stamp: "Formatted at 2026-06-19 13:14:16 on dist-test-slave-mkt3"
server_key: "fd416b141f03bae6a486161934a88c19"
server_key_iv: "22980f447c11811b751d694399cb0f2e"
server_key_version: "encryptionkey@0"
I20260619 13:14:16.239873 6492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe"
format_stamp: "Formatted at 2026-06-19 13:14:16 on dist-test-slave-mkt3"
server_key: "fd416b141f03bae6a486161934a88c19"
server_key_iv: "22980f447c11811b751d694399cb0f2e"
server_key_version: "encryptionkey@0"
I20260619 13:14:16.244593 6492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 13:14:16.247717 6507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:16.248919 6492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 13:14:16.249059 6492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe"
format_stamp: "Formatted at 2026-06-19 13:14:16 on dist-test-slave-mkt3"
server_key: "fd416b141f03bae6a486161934a88c19"
server_key_iv: "22980f447c11811b751d694399cb0f2e"
server_key_version: "encryptionkey@0"
I20260619 13:14:16.249228 6492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:16.315781 6492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:16.319609 6492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:16.319859 6492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:16.336090 6492 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45357
I20260619 13:14:16.336122 6558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45357 every 8 connection(s)
I20260619 13:14:16.337366 6492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:14:16.340903 6559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:16.347265 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 6492
I20260619 13:14:16.347463 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:14:16.348234 388 external_mini_cluster.cc:1428] Setting key d76b413e352990cc8eac3c331e82a633
I20260619 13:14:16.351145 6559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe: Bootstrap starting.
I20260619 13:14:16.354278 6559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:16.355440 6559 log.cc:824] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:16.358536 6559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe: No bootstrap required, opened a new log
Jun 19 13:14:16 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:14:16.366662 6559 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } }
I20260619 13:14:16.367080 6559 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:16.367182 6559 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e798ed377b4da7ab9a1fbdc08cf6fe, State: Initialized, Role: FOLLOWER
I20260619 13:14:16.367723 6559 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } }
I20260619 13:14:16.367929 6559 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:14:16.368062 6559 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:14:16.368206 6559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:16.370379 6559 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } }
I20260619 13:14:16.370482 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:16.350166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40594 (local address 127.0.97.62:45357)
0619 13:14:16.350903 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:16.350914 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:16.350952 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:14:16.352618 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:16.352639 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:16.353075 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:16.353407 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:16.355022 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:16.356847 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:16.358474 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:16.359156 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:16.359286 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:16.363610 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:16.363618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:16.363634 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:16.363669 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:16.366543 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:16.367251 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:16.367260 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:16.367268 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:16.367437 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:16.367806 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:16.367811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:16.367815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:16.368423 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:16.368707 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:16.369124 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 13:14:16.369499 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":310,"threads_started":1}
I20260619 13:14:16.370838 6559 leader_election.cc:302] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3e798ed377b4da7ab9a1fbdc08cf6fe; no voters:
I20260619 13:14:16.371433 6559 leader_election.cc:288] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:14:16.371531 6564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:16.372529 6564 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [term 1 LEADER]: Becoming Leader. State: Replica: a3e798ed377b4da7ab9a1fbdc08cf6fe, State: Running, Role: LEADER
I20260619 13:14:16.373118 6564 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } }
I20260619 13:14:16.373651 6559 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:16.376138 6566 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3e798ed377b4da7ab9a1fbdc08cf6fe. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } }
I20260619 13:14:16.376478 6566 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:16.376657 6565 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } }
I20260619 13:14:16.376963 6565 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:16.378315 6573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:14:16.382509 6573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:14:16.389972 6573 catalog_manager.cc:1347] Generated new cluster ID: 260ce7232d764c108a9ed51331674457
I20260619 13:14:16.390084 6573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:14:16.398788 6573 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:14:16.400470 6573 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:14:16.415648 6573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe: Generated new TSK 0
I20260619 13:14:16.416669 6573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 13:14:16.466961 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45357
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:16.771606 6585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:16.771874 6585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:16.772010 6585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:16.778970 6585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:16.779076 6585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:16.779222 6585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:14:16.788406 6585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:16.789587 6585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:16.790860 6585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:16.802831 6590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:16.802791 6591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:16.802801 6593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:16.803771 6585 server_base.cc:1034] running on GCE node
I20260619 13:14:16.804327 6585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:16.805178 6585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:16.806365 6585 hybrid_clock.cc:648] HybridClock initialized: now 1781874856806309 us; error 51 us; skew 500 ppm
I20260619 13:14:16.806644 6585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:16 dist-test-slave-mkt3 krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:16.809430 6585 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 13:14:16.810747 6585 webserver.cc:460] Webserver started at http://127.0.97.1:45729/ using document root <none> and password file <none>
I20260619 13:14:16.811368 6585 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:16.811478 6585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:16.811746 6585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:16.814435 6585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "d085b70f4c4d4185be1b6a426e622164"
format_stamp: "Formatted at 2026-06-19 13:14:16 on dist-test-slave-mkt3"
server_key: "bed568b79e0e35ef53aac9b96f77436a"
server_key_iv: "7839f7cf1b878c5d7ea3029f494b61ac"
server_key_version: "encryptionkey@0"
I20260619 13:14:16.815135 6585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d085b70f4c4d4185be1b6a426e622164"
format_stamp: "Formatted at 2026-06-19 13:14:16 on dist-test-slave-mkt3"
server_key: "bed568b79e0e35ef53aac9b96f77436a"
server_key_iv: "7839f7cf1b878c5d7ea3029f494b61ac"
server_key_version: "encryptionkey@0"
I20260619 13:14:16.819775 6585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:14:16.823028 6600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:16.824227 6585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:14:16.824365 6585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "d085b70f4c4d4185be1b6a426e622164"
format_stamp: "Formatted at 2026-06-19 13:14:16 on dist-test-slave-mkt3"
server_key: "bed568b79e0e35ef53aac9b96f77436a"
server_key_iv: "7839f7cf1b878c5d7ea3029f494b61ac"
server_key_version: "encryptionkey@0"
I20260619 13:14:16.824527 6585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:16.846374 6585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:16.849712 6585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:16.849980 6585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:16.850814 6585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:16.852010 6585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:16.852080 6585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:16.852188 6585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:16.852238 6585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:16.880151 6585 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43431
I20260619 13:14:16.880208 6712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43431 every 8 connection(s)
I20260619 13:14:16.881363 6585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:14:16.885589 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 6585
I20260619 13:14:16.885792 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:14:16.886569 388 external_mini_cluster.cc:1428] Setting key 94ff429db4241fc57980e393455d6940
Jun 19 13:14:16 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:14:16.901435 6715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:16.883715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46325 (local address 127.0.97.62:45357)
0619 13:14:16.884105 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:16.884109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:16.885098 (+ 989us) server_negotiation.cc:407] Connection header received
0619 13:14:16.887470 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:16.887474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:16.887540 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:16.887645 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:16.889355 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:16.890108 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:16.892067 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:16.892443 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:16.892532 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:16.896360 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:16.896365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:16.896368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:16.896398 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:16.898674 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:16.899437 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:16.899440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:16.899443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:16.899546 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:16.899936 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:16.899939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:16.899941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:16.900176 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:16.900292 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:16.901010 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 13:14:16.901178 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":192,"threads_started":1}
I20260619 13:14:16.902896 6716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:16.884293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.1:46325)
0619 13:14:16.884946 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:16.884982 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:16.887187 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:16.887842 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:16.887850 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:16.888425 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:16.889175 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:16.889200 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:16.890263 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:16.890268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:16.891948 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:16.891962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:16.892637 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:16.892643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:16.892891 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:16.893536 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:16.893561 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:16.896156 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:16.898794 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:16.898803 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:16.898819 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:16.899344 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:16.899630 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:16.899634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:16.899637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:16.899848 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:16.900280 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:16.900287 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:16.900658 (+ 371us) client_negotiation.cc:769] Sending connection context
0619 13:14:16.900968 (+ 310us) client_negotiation.cc:240] Negotiation successful
0619 13:14:16.901476 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":240,"threads_started":1}
I20260619 13:14:16.904373 6713 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45357
I20260619 13:14:16.904668 6713 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:16.905359 6713 heartbeater.cc:511] Master 127.0.97.62:45357 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:14:16.907400 6524 ts_manager.cc:194] Registered new tserver with Master: d085b70f4c4d4185be1b6a426e622164 (127.0.97.1:43431)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260619 13:14:16.909157 6524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260619 13:14:16.928803 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45357
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:17.231936 6719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:17.232216 6719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:17.232339 6719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:17.239398 6719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:17.239514 6719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:17.239677 6719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:14:17.249128 6719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:17.250368 6719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:17.251585 6719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:17.263383 6725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:17.263404 6724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:17.263563 6727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:17.264137 6719 server_base.cc:1034] running on GCE node
I20260619 13:14:17.264669 6719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:17.265580 6719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:17.266772 6719 hybrid_clock.cc:648] HybridClock initialized: now 1781874857266729 us; error 33 us; skew 500 ppm
I20260619 13:14:17.267052 6719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:17 dist-test-slave-mkt3 krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874857, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:17.269886 6719 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 13:14:17.271193 6719 webserver.cc:460] Webserver started at http://127.0.97.2:43357/ using document root <none> and password file <none>
I20260619 13:14:17.271821 6719 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:17.271930 6719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:17.272192 6719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:17.274994 6719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "436161bbf1444a99a6611cda7569a9a8"
format_stamp: "Formatted at 2026-06-19 13:14:17 on dist-test-slave-mkt3"
server_key: "a9e2df996b49a933b755ffbb9fbc90d7"
server_key_iv: "55ecfd1cec22c0b0fd6ea372c29b0464"
server_key_version: "encryptionkey@0"
I20260619 13:14:17.275754 6719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "436161bbf1444a99a6611cda7569a9a8"
format_stamp: "Formatted at 2026-06-19 13:14:17 on dist-test-slave-mkt3"
server_key: "a9e2df996b49a933b755ffbb9fbc90d7"
server_key_iv: "55ecfd1cec22c0b0fd6ea372c29b0464"
server_key_version: "encryptionkey@0"
I20260619 13:14:17.280552 6719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:14:17.283706 6734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:17.284907 6719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 13:14:17.285046 6719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "436161bbf1444a99a6611cda7569a9a8"
format_stamp: "Formatted at 2026-06-19 13:14:17 on dist-test-slave-mkt3"
server_key: "a9e2df996b49a933b755ffbb9fbc90d7"
server_key_iv: "55ecfd1cec22c0b0fd6ea372c29b0464"
server_key_version: "encryptionkey@0"
I20260619 13:14:17.285228 6719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:17.304155 6719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:17.307816 6719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:17.308068 6719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:17.308919 6719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:17.310122 6719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:17.310197 6719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:17.310312 6719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:17.310360 6719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:17.338402 6719 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36241
I20260619 13:14:17.338433 6846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36241 every 8 connection(s)
I20260619 13:14:17.339793 6719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:14:17.346359 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 6719
I20260619 13:14:17.346532 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:14:17.347078 388 external_mini_cluster.cc:1428] Setting key 83c8f5b3416383199d7fd591b596bafd
Jun 19 13:14:17 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874857, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:14:17.360571 6715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:17.342134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41145 (local address 127.0.97.62:45357)
0619 13:14:17.342330 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:17.342337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:17.343576 (+ 1239us) server_negotiation.cc:407] Connection header received
0619 13:14:17.345247 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:17.345252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:17.345349 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:17.345467 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:17.347256 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:17.348368 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:17.350626 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:17.351164 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:17.351278 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:17.354985 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:17.354991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:17.354996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:17.355032 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:17.357946 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:17.358584 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:17.358590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:17.358594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:17.358733 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:17.359070 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:17.359074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:17.359077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:17.359365 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:17.359492 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:17.360069 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 13:14:17.360281 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:14:17.361341 6849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:17.342651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.2:41145)
0619 13:14:17.343399 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:17.343450 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:17.345021 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:17.345635 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:17.345643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:17.346289 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:17.347043 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:17.347069 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:17.348571 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:17.348576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:17.350481 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:17.350495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:17.351364 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:17.351371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:17.351634 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:17.352329 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:17.352361 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:17.354778 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:17.358070 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:17.358078 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:17.358091 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:17.358477 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:17.358819 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:17.358822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:17.358824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:17.358982 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:17.359459 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:17.359464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:17.359739 (+ 275us) client_negotiation.cc:769] Sending connection context
0619 13:14:17.359956 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 13:14:17.360300 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":295,"threads_started":1}
I20260619 13:14:17.362697 6847 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45357
I20260619 13:14:17.362974 6847 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:17.363626 6847 heartbeater.cc:511] Master 127.0.97.62:45357 requested a full tablet report, sending...
I20260619 13:14:17.364953 6524 ts_manager.cc:194] Registered new tserver with Master: 436161bbf1444a99a6611cda7569a9a8 (127.0.97.2:36241)
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:14:17.365728 6524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 13:14:17.386973 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45357
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:17.684466 6852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:17.684723 6852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:17.684840 6852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:17.691891 6852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:17.691998 6852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:17.692133 6852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:14:17.701391 6852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:17.702620 6852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:17.703863 6852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:17.715727 6857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:17.715757 6860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:17.715692 6858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:17.716403 6852 server_base.cc:1034] running on GCE node
I20260619 13:14:17.716949 6852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:17.717804 6852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:17.719012 6852 hybrid_clock.cc:648] HybridClock initialized: now 1781874857718969 us; error 41 us; skew 500 ppm
I20260619 13:14:17.719290 6852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:17 dist-test-slave-mkt3 krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874857, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:17.722064 6852 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 13:14:17.723349 6852 webserver.cc:460] Webserver started at http://127.0.97.3:41725/ using document root <none> and password file <none>
I20260619 13:14:17.723942 6852 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:17.724049 6852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:17.724320 6852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:17.727028 6852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "410babf59db6434c9d72b772ac83f5f9"
format_stamp: "Formatted at 2026-06-19 13:14:17 on dist-test-slave-mkt3"
server_key: "35d6c803bdd6ea0d513f6606840b6f06"
server_key_iv: "290766c59bdd787ce82825eb6bad80a8"
server_key_version: "encryptionkey@0"
I20260619 13:14:17.727784 6852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "410babf59db6434c9d72b772ac83f5f9"
format_stamp: "Formatted at 2026-06-19 13:14:17 on dist-test-slave-mkt3"
server_key: "35d6c803bdd6ea0d513f6606840b6f06"
server_key_iv: "290766c59bdd787ce82825eb6bad80a8"
server_key_version: "encryptionkey@0"
I20260619 13:14:17.732422 6852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:14:17.735606 6867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:17.736788 6852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 13:14:17.736919 6852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "410babf59db6434c9d72b772ac83f5f9"
format_stamp: "Formatted at 2026-06-19 13:14:17 on dist-test-slave-mkt3"
server_key: "35d6c803bdd6ea0d513f6606840b6f06"
server_key_iv: "290766c59bdd787ce82825eb6bad80a8"
server_key_version: "encryptionkey@0"
I20260619 13:14:17.737075 6852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:17.755985 6852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:17.759809 6852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:17.760056 6852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:17.760921 6852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:17.762176 6852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:17.762251 6852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:17.762362 6852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:17.762420 6852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:17.791085 6852 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45933
I20260619 13:14:17.791146 6979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45933 every 8 connection(s)
I20260619 13:14:17.792248 6852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:14:17.802089 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 6852
I20260619 13:14:17.802263 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:14:17.802773 388 external_mini_cluster.cc:1428] Setting key 1ffce22997fcc0277b154c2cae21452c
Jun 19 13:14:17 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874857, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:14:17.809883 6715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:17.794474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48981 (local address 127.0.97.62:45357)
0619 13:14:17.794677 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:17.794684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:17.795816 (+ 1132us) server_negotiation.cc:407] Connection header received
0619 13:14:17.797458 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:17.797462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:17.797527 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:17.797610 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:17.799032 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:17.799755 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:17.801222 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:17.801604 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:17.801689 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:17.805234 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:17.805240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:17.805245 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:17.805287 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:17.807482 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:17.808116 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:17.808119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:17.808121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:17.808249 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:17.808576 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:17.808579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:17.808581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:17.808839 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:17.808942 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:17.809485 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 13:14:17.809633 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:14:17.810801 6982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:17.794972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.3:48981)
0619 13:14:17.795656 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:17.795698 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:17.797226 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:17.797792 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:17.797801 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:17.798351 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:17.798886 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:17.798903 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:17.799877 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:17.799881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:17.801095 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:17.801106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:17.801808 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:17.801816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:17.802105 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:17.802766 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:17.802794 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:17.805027 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:17.807615 (+ 2588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:17.807622 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:17.807633 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:17.808020 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:17.808326 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:17.808328 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:17.808331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:17.808496 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:17.808923 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:17.808928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:17.809217 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 13:14:17.809426 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 13:14:17.809775 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":274,"threads_started":1}
I20260619 13:14:17.812006 6980 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45357
I20260619 13:14:17.812273 6980 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:17.812919 6980 heartbeater.cc:511] Master 127.0.97.62:45357 requested a full tablet report, sending...
I20260619 13:14:17.814256 6524 ts_manager.cc:194] Registered new tserver with Master: 410babf59db6434c9d72b772ac83f5f9 (127.0.97.3:45933)
I20260619 13:14:17.814973 6524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48981
I20260619 13:14:17.817562 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260619 13:14:17.856765 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45041
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45357,127.0.97.61:45041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:14:17.911996 6713 heartbeater.cc:503] Master 127.0.97.62:45357 was elected leader, sending a full tablet report...
W20260619 13:14:18.170116 6985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:18.170383 6985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:18.170509 6985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:18.177562 6985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:18.177670 6985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:18.177722 6985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:18.177811 6985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:18.177870 6985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:14:18.187357 6985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45357,127.0.97.61:45041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
W20260619 13:14:18.187674 6985 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:45357,127.0.97.61:45041'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 13:14:18.188694 6985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:18.189976 6985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:18.200548 6990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:18.200523 6991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:18.200711 6993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:18.201558 6985 server_base.cc:1034] running on GCE node
I20260619 13:14:18.202273 6985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:18.203537 6985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:18.204744 6985 hybrid_clock.cc:648] HybridClock initialized: now 1781874858204674 us; error 48 us; skew 500 ppm
I20260619 13:14:18.205027 6985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:18 dist-test-slave-mkt3 krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874858, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:18.207890 6985 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260619 13:14:18.209172 6985 webserver.cc:460] Webserver started at http://127.0.97.61:40143/ using document root <none> and password file <none>
I20260619 13:14:18.209820 6985 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:18.209947 6985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:18.210238 6985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:18.213027 6985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/data/instance:
uuid: "5c7b3f0d5a26443f84775a18058284b0"
format_stamp: "Formatted at 2026-06-19 13:14:18 on dist-test-slave-mkt3"
server_key: "b40177a88d85897d14e177ae88a8e768"
server_key_iv: "2d04d2c1a5ab6d0e4d7f79b5f0edba68"
server_key_version: "encryptionkey@0"
I20260619 13:14:18.213774 6985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/wal/instance:
uuid: "5c7b3f0d5a26443f84775a18058284b0"
format_stamp: "Formatted at 2026-06-19 13:14:18 on dist-test-slave-mkt3"
server_key: "b40177a88d85897d14e177ae88a8e768"
server_key_iv: "2d04d2c1a5ab6d0e4d7f79b5f0edba68"
server_key_version: "encryptionkey@0"
I20260619 13:14:18.218474 6985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 13:14:18.221738 7000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:18.223016 6985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:14:18.223163 6985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/wal
uuid: "5c7b3f0d5a26443f84775a18058284b0"
format_stamp: "Formatted at 2026-06-19 13:14:18 on dist-test-slave-mkt3"
server_key: "b40177a88d85897d14e177ae88a8e768"
server_key_iv: "2d04d2c1a5ab6d0e4d7f79b5f0edba68"
server_key_version: "encryptionkey@0"
I20260619 13:14:18.223347 6985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:18.236979 6985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:18.240500 6985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:18.240721 6985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:18.255930 6985 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:45041
I20260619 13:14:18.255951 7051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:45041 every 8 connection(s)
I20260619 13:14:18.257108 6985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/data/info.pb
I20260619 13:14:18.260488 7052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:18.267030 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 6985
I20260619 13:14:18.266543 7052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:14:18 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874858, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 13:14:18 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 13:14:18.287709 6715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.269093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.0.97.62:45357)
0619 13:14:18.269311 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:18.269317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:18.270107 (+ 790us) server_negotiation.cc:407] Connection header received
0619 13:14:18.271831 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:18.271836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:18.271910 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:18.272018 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:18.273951 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.275066 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.277085 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.277711 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.277891 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.281642 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:18.281647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:18.281650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:18.281682 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:18.284568 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.285371 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.285376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.285379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.285553 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.285991 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.285994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.285996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.286350 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:18.286488 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:18.287224 (+ 736us) server_negotiation.cc:299] Negotiation successful
0619 13:14:18.287400 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:14:18.288776 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.269099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:40600)
0619 13:14:18.269946 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:18.269997 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:18.271562 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:18.272475 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:18.272482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:18.273253 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:18.273780 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.273800 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.275185 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.275189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.276962 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.276974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.277973 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.277982 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.278242 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.278987 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:18.279015 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:18.281464 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:18.284680 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.284689 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.284702 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.285237 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.285649 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.285653 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.285657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.285897 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.286479 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:18.286487 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:18.286874 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 13:14:18.287110 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 13:14:18.287598 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":41,"thread_start_us":364,"threads_started":1}
I20260619 13:14:18.288808 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.269058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39642 (local address 127.0.97.61:45041)
0619 13:14:18.269971 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:18.269982 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:18.270018 (+ 36us) server_negotiation.cc:407] Connection header received
0619 13:14:18.271555 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:18.271578 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:18.271853 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:18.272057 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:18.273779 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.275408 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.277081 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.277925 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.278116 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.282267 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:18.282276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:18.282288 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:18.282315 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:18.285475 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.286216 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.286225 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.286228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.286370 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.286707 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.286711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.286713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.287022 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:18.287274 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:18.287594 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 13:14:18.287966 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":702,"thread_start_us":468,"threads_started":1}
I20260619 13:14:18.290527 7052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:14:18 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874858, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 13:14:18.301266 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.291097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45041 (local address 127.0.0.1:39656)
0619 13:14:18.291251 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:18.291266 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:18.291336 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:18.291666 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:18.291670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:18.291974 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:18.292094 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.292101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.292975 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.292980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.293803 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.293813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.294460 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.294466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.294624 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.295056 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:18.295070 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:18.296810 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:18.299404 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.299408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.299410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.299732 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.300039 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.300042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.300044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.300153 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.300575 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:18.300578 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:18.300685 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 13:14:18.300787 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 13:14:18.300972 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 13:14:18.301322 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.291284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.0.97.61:45041)
0619 13:14:18.291428 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:18.291433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:18.291449 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:14:18.291488 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:18.291492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:18.291551 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:18.291638 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:18.292207 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.292867 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.293943 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.294334 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.294441 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.296990 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:18.296994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:18.296997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:18.297024 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:18.299299 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.299837 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.299841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.299843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.299953 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.300251 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.300254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.300256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.300491 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:18.300588 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:18.300850 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 13:14:18.301089 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":42}
I20260619 13:14:18.305656 7052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: Bootstrap starting.
I20260619 13:14:18.308719 7052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:18.309700 7052 log.cc:824] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:18.312161 7052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: No bootstrap required, opened a new log
I20260619 13:14:18.318840 7052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } }
I20260619 13:14:18.319172 7052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:18.319275 7052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7b3f0d5a26443f84775a18058284b0, State: Initialized, Role: FOLLOWER
I20260619 13:14:18.319813 7052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } }
I20260619 13:14:18.320533 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } } }
I20260619 13:14:18.320802 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:14:18.321523 7052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 13:14:18.329641 7069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:14:18.329779 7069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:14:18.331988 6985 master_runner.cc:428] Detected that this master 5c7b3f0d5a26443f84775a18058284b0 is joining an existing cluster
I20260619 13:14:18.332079 6985 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:45041
I20260619 13:14:18.361414 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.335436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39666 (local address 127.0.97.61:45041)
0619 13:14:18.335630 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:18.335635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:18.345000 (+ 9365us) server_negotiation.cc:407] Connection header received
0619 13:14:18.345228 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:18.345234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:18.345308 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:18.345382 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:18.346833 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.347957 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.350565 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.351361 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.351461 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.354021 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:18.354027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:18.354032 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:18.354064 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:18.358106 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.358864 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.358869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.358872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.359011 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.359668 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.359673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.359676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.360020 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:18.360168 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:18.360644 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 13:14:18.361035 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":62}
I20260619 13:14:18.362094 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.338035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45041 (local address 127.0.0.1:39666)
0619 13:14:18.344894 (+ 6859us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:18.344920 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:18.345098 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:18.345572 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:18.345577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:18.346310 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:18.346675 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.346687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.348174 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.348179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.349705 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.349720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.351784 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.351792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.352080 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.352740 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:18.352772 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:18.353829 (+ 1057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:18.358226 (+ 4397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.358230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.358234 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.358727 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.359400 (+ 673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.359404 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.359408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.359561 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.360231 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:18.360235 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:18.360370 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 13:14:18.360510 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 13:14:18.361658 (+ 1148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6691,"thread_start_us":11842,"threads_started":1}
I20260619 13:14:18.364074 6715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.335676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.0.97.62:45357)
0619 13:14:18.335854 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:18.335859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:18.350409 (+ 14550us) server_negotiation.cc:407] Connection header received
0619 13:14:18.350523 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:18.350528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:18.350616 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:18.350706 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:18.352106 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.353190 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.355237 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.355895 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.355980 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.358385 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:18.358391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:18.358395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:18.358430 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:18.361394 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.362109 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.362114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.362117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.362242 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.362615 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.362618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.362621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.362949 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:18.363066 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:18.363575 (+ 509us) server_negotiation.cc:299] Negotiation successful
0619 13:14:18.363779 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:14:18.364914 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.338231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:40610)
0619 13:14:18.350288 (+ 12057us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:18.350308 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:18.350427 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:18.350868 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:18.350872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:18.351535 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:18.351943 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.351958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.353390 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.353395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.355089 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.355103 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.356228 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.356233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.356407 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.356958 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:18.356981 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:18.358213 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:18.361528 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.361532 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.361536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.361985 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.362362 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.362366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.362369 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.362503 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.363133 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:18.363137 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:18.363317 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 13:14:18.364230 (+ 913us) client_negotiation.cc:240] Negotiation successful
0619 13:14:18.364520 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11915,"thread_start_us":11716,"threads_started":1}
I20260619 13:14:18.368418 6847 heartbeater.cc:503] Master 127.0.97.62:45357 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Jun 19 13:14:18 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 13:14:18.387343 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.371213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.0.97.61:45041)
0619 13:14:18.371958 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:18.371964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:18.371982 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:14:18.372069 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:18.372074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:18.372175 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:18.372274 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:18.373572 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.374693 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.376280 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.376972 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.377143 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.380876 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:18.380883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:18.380889 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:18.380926 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:18.384417 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.385195 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.385201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.385205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.385374 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.385789 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.385794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.385798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.386154 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:18.386295 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:18.386752 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 13:14:18.386997 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637}
I20260619 13:14:18.387378 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.370825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45041 (local address 127.0.0.1:39670)
0619 13:14:18.371450 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:18.371493 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:18.371639 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:18.372344 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:18.372352 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:18.373106 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:18.373364 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.373377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.374840 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.374844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.376069 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.376084 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.377203 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.377210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.377399 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.378135 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:18.378155 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:18.380678 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:18.384557 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.384569 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.384576 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.385079 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.385478 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.385481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.385484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.385666 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.386272 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:18.386279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:18.386460 (+ 181us) client_negotiation.cc:769] Sending connection context
0619 13:14:18.386650 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 13:14:18.386985 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":432,"threads_started":1}
I20260619 13:14:18.389735 6524 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:45041: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 45041
}
attrs {
promote: true
}
}
dest_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe"
cas_config_opid_index: -1
I20260619 13:14:18.390930 6524 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: true } }
I20260619 13:14:18.394500 7081 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5c7b3f0d5a26443f84775a18058284b0 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: true } } }
I20260619 13:14:18.397485 7027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:18.398291 7081 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:45041
I20260619 13:14:18.398695 7081 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: true } } }
I20260619 13:14:18.398882 7081 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:18.399031 6985 master.cc:537] Master@127.0.97.61:45041 shutting down...
I20260619 13:14:18.403326 7027 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 FOLLOWER]: Refusing update from remote peer a3e798ed377b4da7ab9a1fbdc08cf6fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260619 13:14:18.405365 7081 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:18.405665 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42585
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45357,127.0.97.61:45041,127.0.97.60:42585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:18.408717 6511 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 13:14:18.416127 6985 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:18.416536 6985 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:18.416723 6985 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: stopping tablet replica
I20260619 13:14:18.423857 6985 master.cc:552] Master@127.0.97.61:45041 shutdown complete.
I20260619 13:14:18.423956 6985 master_runner.cc:305] Clearing existing system tablet
I20260619 13:14:18.424405 6985 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:18.424506 6985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:18.425467 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.425006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45041 (local address 127.0.0.1:39686)
0619 13:14:18.425178 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:18.425289 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 13:14:18.425940 6511 consensus_peers.cc:487] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe -> Peer 5c7b3f0d5a26443f84775a18058284b0 (127.0.97.61:45041): Couldn't send request to peer 5c7b3f0d5a26443f84775a18058284b0. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 13:14:18.427520 6985 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 13:14:18.429292 7087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:18.429620 6985 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:18.429955 6985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 13:14:18.430053 6985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/wal
uuid: "5c7b3f0d5a26443f84775a18058284b0"
format_stamp: "Formatted at 2026-06-19 13:14:18 on dist-test-slave-mkt3"
server_key: "b40177a88d85897d14e177ae88a8e768"
server_key_iv: "2d04d2c1a5ab6d0e4d7f79b5f0edba68"
server_key_version: "encryptionkey@0"
I20260619 13:14:18.435037 6985 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:18.441670 6985 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 13:14:18.441854 6985 log.cc:1196] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 13:14:18.442256 6985 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: Deleting consensus metadata
I20260619 13:14:18.442494 6985 master_runner.cc:315] Copying system tablet from 127.0.97.62:45357
I20260619 13:14:18.451327 6985 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:45357
I20260619 13:14:18.467862 6715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.452083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40614 (local address 127.0.97.62:45357)
0619 13:14:18.452271 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:18.452278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:18.454406 (+ 2128us) server_negotiation.cc:407] Connection header received
0619 13:14:18.454603 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:18.454609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:18.454694 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:18.454791 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:18.455770 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.456829 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.458586 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.459262 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.459397 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.461604 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:18.461611 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:18.461617 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:18.461656 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:18.464959 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.465737 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.465743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.465747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.465911 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.466338 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.466343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.466346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.466681 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:18.466824 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:18.467294 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 13:14:18.467517 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:14:18.467993 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.452091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:40614)
0619 13:14:18.454282 (+ 2191us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:18.454303 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:18.454459 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:18.454826 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:18.454831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:18.455348 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:18.455612 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.455628 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.456990 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.456995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.458439 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.458454 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.459435 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.459442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.459631 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.460340 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:18.460362 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:18.461402 (+ 1040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:18.465106 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.465111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.465114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.465618 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.466029 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.466035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.466039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.466226 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.466817 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:18.466822 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:18.467030 (+ 208us) client_negotiation.cc:769] Sending connection context
0619 13:14:18.467193 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 13:14:18.467530 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2040,"thread_start_us":1920,"threads_started":1}
I20260619 13:14:18.469156 6544 tablet_copy_service.cc:140] P a3e798ed377b4da7ab9a1fbdc08cf6fe: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5c7b3f0d5a26443f84775a18058284b0 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40614)
I20260619 13:14:18.469458 6544 tablet_copy_service.cc:161] P a3e798ed377b4da7ab9a1fbdc08cf6fe: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5c7b3f0d5a26443f84775a18058284b0 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40614: session id = 5c7b3f0d5a26443f84775a18058284b0-00000000000000000000000000000000
I20260619 13:14:18.473664 6544 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 13:14:18.475839 6985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 13:14:18.482311 6985 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: tablet copy: Starting download of 0 data blocks...
I20260619 13:14:18.482591 6985 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: tablet copy: Starting download of 1 WAL segments...
I20260619 13:14:18.485100 6985 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 13:14:18.489737 6544 tablet_copy_service.cc:342] P a3e798ed377b4da7ab9a1fbdc08cf6fe: Request end of tablet copy session 5c7b3f0d5a26443f84775a18058284b0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40614
I20260619 13:14:18.489948 6544 tablet_copy_service.cc:433] P a3e798ed377b4da7ab9a1fbdc08cf6fe: ending tablet copy session 5c7b3f0d5a26443f84775a18058284b0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5c7b3f0d5a26443f84775a18058284b0
I20260619 13:14:18.501406 6985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:18.503957 7094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:18.505128 7095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:18.509511 6985 server_base.cc:1034] running on GCE node
W20260619 13:14:18.509857 7097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:18.510270 6985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:18.510893 6985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:18.512099 6985 hybrid_clock.cc:648] HybridClock initialized: now 1781874858512062 us; error 63 us; skew 500 ppm
I20260619 13:14:18.512319 6985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:18 dist-test-slave-mkt3 krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874858, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:18.514564 6985 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260619 13:14:18.518688 6985 webserver.cc:460] Webserver started at http://127.0.97.61:34453/ using document root <none> and password file <none>
I20260619 13:14:18.519037 6985 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:18.519120 6985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:18.522012 6985 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 13:14:18.523959 7104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:18.524406 6985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 13:14:18.524552 6985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/wal
uuid: "5c7b3f0d5a26443f84775a18058284b0"
format_stamp: "Formatted at 2026-06-19 13:14:18 on dist-test-slave-mkt3"
server_key: "b40177a88d85897d14e177ae88a8e768"
server_key_iv: "2d04d2c1a5ab6d0e4d7f79b5f0edba68"
server_key_version: "encryptionkey@0"
I20260619 13:14:18.524708 6985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:18.539083 6985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:18.542924 6985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:18.565105 6985 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:45041
I20260619 13:14:18.565152 7155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:45041 every 8 connection(s)
I20260619 13:14:18.566164 6985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-1/data/info.pb
I20260619 13:14:18.571517 7156 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:14:18.575614 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: Bootstrap starting.
I20260619 13:14:18.591871 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:14:18.592363 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: Bootstrap complete.
I20260619 13:14:18.593353 7156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: true } }
I20260619 13:14:18.593604 7156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5c7b3f0d5a26443f84775a18058284b0, State: Initialized, Role: LEARNER
I20260619 13:14:18.593863 7156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: true } }
I20260619 13:14:18.594655 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: true } } }
I20260619 13:14:18.595080 7156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:18.595113 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: This master's current role is: LEARNER
I20260619 13:14:18.605346 7170 catalog_manager.cc:1259] Loaded cluster ID: 260ce7232d764c108a9ed51331674457
I20260619 13:14:18.605441 7170 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: loading cluster ID for follower catalog manager: success
I20260619 13:14:18.609441 7170 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: acquiring CA information for follower catalog manager: success
I20260619 13:14:18.612051 7170 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 13:14:18.778733 7085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:18.778995 7085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:18.779125 7085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:18.786487 7085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:18.786578 7085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:18.786644 7085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:18.786713 7085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:18.786767 7085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:14:18.798555 7085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45357,127.0.97.61:45041,127.0.97.60:42585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:42585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:18.799819 7085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:18.800972 7085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:18.811980 7175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:18.811902 7178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:18.811934 7176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:18.812621 7085 server_base.cc:1034] running on GCE node
I20260619 13:14:18.813457 7085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:18.814785 7085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:18.815989 7085 hybrid_clock.cc:648] HybridClock initialized: now 1781874858815935 us; error 43 us; skew 500 ppm
I20260619 13:14:18.816303 7085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:18.817530 6980 heartbeater.cc:503] Master 127.0.97.62:45357 was elected leader, sending a full tablet report...
Jun 19 13:14:18 dist-test-slave-mkt3 krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874858, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:18.819808 7085 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260619 13:14:18.821069 7085 webserver.cc:460] Webserver started at http://127.0.97.60:45337/ using document root <none> and password file <none>
I20260619 13:14:18.821671 7085 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:18.821782 7085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:18.822062 7085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:18.824760 7085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/data/instance:
uuid: "13c24fa7041b4956a8601b23968dd506"
format_stamp: "Formatted at 2026-06-19 13:14:18 on dist-test-slave-mkt3"
server_key: "5ad70227fe37f7807f5f5669b4bab1c5"
server_key_iv: "e9a7b2ef871174840597d410da102fd6"
server_key_version: "encryptionkey@0"
I20260619 13:14:18.825645 7085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/wal/instance:
uuid: "13c24fa7041b4956a8601b23968dd506"
format_stamp: "Formatted at 2026-06-19 13:14:18 on dist-test-slave-mkt3"
server_key: "5ad70227fe37f7807f5f5669b4bab1c5"
server_key_iv: "e9a7b2ef871174840597d410da102fd6"
server_key_version: "encryptionkey@0"
I20260619 13:14:18.830402 7085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:14:18.833459 7185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:18.834656 7085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 13:14:18.834796 7085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/wal
uuid: "13c24fa7041b4956a8601b23968dd506"
format_stamp: "Formatted at 2026-06-19 13:14:18 on dist-test-slave-mkt3"
server_key: "5ad70227fe37f7807f5f5669b4bab1c5"
server_key_iv: "e9a7b2ef871174840597d410da102fd6"
server_key_version: "encryptionkey@0"
I20260619 13:14:18.834955 7085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:18.851162 7085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:18.854807 7085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:18.855037 7085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:18.870908 7085 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:42585
I20260619 13:14:18.871340 7236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:42585 every 8 connection(s)
I20260619 13:14:18.872296 7085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/data/info.pb
I20260619 13:14:18.875751 7237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:18.879436 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 7085
I20260619 13:14:18.882802 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:14:18 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874858, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 13:14:18 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 13:14:18.899852 6715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.885422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.0.97.62:45357)
0619 13:14:18.885587 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:18.885593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:18.886034 (+ 441us) server_negotiation.cc:407] Connection header received
0619 13:14:18.886363 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:18.886369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:18.886442 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:18.886522 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:18.887797 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.888591 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.890074 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.890474 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.890581 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.893793 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:18.893798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:18.893801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:18.893832 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:18.896783 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.897508 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.897513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.897516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.897633 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.898067 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.898072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.898076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.898399 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:18.898527 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:18.899237 (+ 710us) server_negotiation.cc:299] Negotiation successful
0619 13:14:18.899505 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:14:18.900957 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.881478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38106 (local address 127.0.97.60:42585)
0619 13:14:18.883316 (+ 1838us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:18.883332 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:18.883609 (+ 277us) server_negotiation.cc:407] Connection header received
0619 13:14:18.885586 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:18.885612 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:18.886117 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:18.886880 (+ 763us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:18.887916 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.889073 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.890322 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.890807 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.890981 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.894983 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:18.894990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:18.895001 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:18.895027 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:18.897598 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.898307 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.898313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.898315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.898416 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.898779 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.898784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.898786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.899196 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:18.899447 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:18.899779 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 13:14:18.899990 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":1601,"thread_start_us":1394,"threads_started":1}
I20260619 13:14:18.900990 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.885308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:40628)
0619 13:14:18.885909 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:18.885946 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:18.886137 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:18.886589 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:18.886595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:18.887294 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:18.887626 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.887642 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.888735 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.888738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.889945 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.889956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.890676 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.890683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.890873 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.891518 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:18.891539 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:18.893612 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:18.896899 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.896910 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.896920 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.897404 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.897750 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.897753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.897756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.897962 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.898514 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:18.898521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:18.898858 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 13:14:18.899143 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 13:14:18.899570 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":20,"thread_start_us":265,"threads_started":1}
I20260619 13:14:18.902669 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:14:18 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874858, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 13:14:18.916353 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.903371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45041 (local address 127.0.0.1:39688)
0619 13:14:18.903521 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:18.903539 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:18.903616 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:18.906421 (+ 2805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:18.906425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:18.906765 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:18.906881 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.906889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.908078 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.908084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.908944 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.908953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.909564 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.909571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.909763 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.910243 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:18.910258 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:18.912014 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:18.914510 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.914515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.914519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.914853 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.915192 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.915196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.915199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.915341 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.915741 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:18.915744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:18.915862 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 13:14:18.915955 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 13:14:18.916140 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 13:14:18.916656 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.903352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.0.97.61:45041)
0619 13:14:18.906176 (+ 2824us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:18.906180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:18.906195 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:14:18.906245 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:18.906248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:18.906298 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:18.906383 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:18.907010 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.907957 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.909065 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.909448 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.909548 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.912172 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:18.912177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:18.912180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:18.912208 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:18.914408 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.914949 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.914954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.914958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.915081 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.915424 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.915427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.915429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.915665 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:18.916151 (+ 486us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:18.916242 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 13:14:18.916376 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2719,"thread_start_us":2621,"threads_started":1}
I20260619 13:14:18.917543 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:14:18 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874858, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 13:14:18.927977 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.918137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42585 (local address 127.0.0.1:38110)
0619 13:14:18.918285 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:18.918299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:18.918378 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:18.918615 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:18.918619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:18.918931 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:18.919041 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.919048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.919890 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.919893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.920666 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.920675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.921260 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.921264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.921407 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.921816 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:18.921829 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:18.923575 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:18.926166 (+ 2591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.926169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.926172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.926515 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.926786 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.926789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.926791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.926916 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.927351 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:18.927354 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:18.927455 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 13:14:18.927553 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 13:14:18.927712 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 13:14:18.928038 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.918292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38110 (local address 127.0.97.60:42585)
0619 13:14:18.918424 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:18.918429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:18.918443 (+ 14us) server_negotiation.cc:407] Connection header received
0619 13:14:18.918477 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:18.918480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:18.918522 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:18.918589 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:18.919150 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.919791 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.920770 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.921153 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.921240 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.923725 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:18.923729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:18.923732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:18.923768 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:18.926071 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.926592 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.926595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.926597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.926703 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.926981 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.926983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.926985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.927262 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:18.927361 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:18.927606 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 13:14:18.927814 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":46}
I20260619 13:14:18.932202 7237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: Bootstrap starting.
I20260619 13:14:18.935127 7237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:18.936457 7237 log.cc:824] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:18.938792 7237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: No bootstrap required, opened a new log
I20260619 13:14:18.945369 7237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } }
I20260619 13:14:18.945665 7237 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:18.945776 7237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c24fa7041b4956a8601b23968dd506, State: Initialized, Role: FOLLOWER
I20260619 13:14:18.946343 7237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } }
I20260619 13:14:18.947204 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } } }
I20260619 13:14:18.947593 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:14:18.948310 7237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 13:14:18.965744 7255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:14:18.966003 7255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:14:18.969214 7085 master_runner.cc:428] Detected that this master 13c24fa7041b4956a8601b23968dd506 is joining an existing cluster
I20260619 13:14:18.969332 7085 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:42585
I20260619 13:14:18.975503 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.968246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.0.97.61:45041)
0619 13:14:18.968426 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:18.968433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:18.968705 (+ 272us) server_negotiation.cc:407] Connection header received
0619 13:14:18.968865 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:18.968870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:18.968964 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:18.969058 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:18.969601 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.970654 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.972868 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.974178 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.974329 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.974532 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:18.975129 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 13:14:18.975262 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 13:14:18.976245 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.968022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45041 (local address 127.0.0.1:39700)
0619 13:14:18.968604 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:18.968629 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:18.968747 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:18.969242 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:18.969247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:18.969274 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:18.969444 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.969456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.970862 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.970867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.972698 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.972713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.974674 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.974681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.974910 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.974983 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 13:14:18.975825 (+ 842us) client_negotiation.cc:240] Negotiation successful
0619 13:14:18.975968 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":410,"threads_started":1}
I20260619 13:14:18.977181 7131 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 13:14:18.977931 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3e798ed377b4da7ab9a1fbdc08cf6fe. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: true } } }
I20260619 13:14:18.978155 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: This master's current role is: LEARNER
I20260619 13:14:18.996083 6715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.973226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40640 (local address 127.0.97.62:45357)
0619 13:14:18.973385 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:18.973391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:18.981364 (+ 7973us) server_negotiation.cc:407] Connection header received
0619 13:14:18.981600 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:18.981606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:18.981676 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:18.981749 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:18.983129 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.984161 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.986175 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.987239 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.987337 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.989750 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:18.989757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:18.989761 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:18.989798 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:18.992990 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.993897 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.993902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.993906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.994054 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.994526 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.994530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.994533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:18.994802 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:18.994894 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:18.995507 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 13:14:18.995714 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:14:18.997462 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.973226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:40640)
0619 13:14:18.981267 (+ 8041us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:18.981291 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:18.981464 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:18.981960 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:18.981965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:18.982612 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:18.982972 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.982985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.984326 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.984331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.986019 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.986033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.987589 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.987595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.987774 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.988470 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:18.988493 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:18.989511 (+ 1018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:18.993148 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.993156 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.993161 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.993715 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.994195 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.994199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.994203 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.994395 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:18.994969 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:18.994972 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:18.995179 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 13:14:18.996314 (+ 1135us) client_negotiation.cc:240] Negotiation successful
0619 13:14:18.996692 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7906,"mutex_wait_us":3,"thread_start_us":12119,"threads_started":1}
I20260619 13:14:19.002844 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.973901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.0.97.61:45041)
0619 13:14:18.975785 (+ 1884us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:18.975790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:18.985755 (+ 9965us) server_negotiation.cc:407] Connection header received
0619 13:14:18.986008 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:18.986012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:18.986081 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:18.986141 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:18.988280 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.989365 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.991442 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.992102 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.992188 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.994731 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:18.994739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:18.994744 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:18.994788 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:18.999087 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:18.999882 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:18.999887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:18.999891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.000036 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.000848 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.000853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.000856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.001190 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:19.001325 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:19.002252 (+ 927us) server_negotiation.cc:299] Negotiation successful
0619 13:14:19.002486 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1780,"thread_start_us":11234,"threads_started":1}
I20260619 13:14:19.004272 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.980591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45041 (local address 127.0.0.1:39704)
0619 13:14:18.985668 (+ 5077us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:18.985684 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:18.985864 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:18.986348 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:18.986353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:18.987707 (+ 1354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:18.988062 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.988074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.989592 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.989597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.991269 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.991285 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.992473 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.992479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.992650 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:18.993222 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:18.993242 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:18.994442 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:18.999250 (+ 4808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:18.999266 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:18.999274 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:18.999756 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:19.000544 (+ 788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:19.000549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:19.000552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:19.000718 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:19.001390 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:19.001395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:19.001960 (+ 565us) client_negotiation.cc:769] Sending connection context
0619 13:14:19.003078 (+ 1118us) client_negotiation.cc:240] Negotiation successful
0619 13:14:19.003795 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4924,"thread_start_us":780,"threads_started":1}
I20260619 13:14:19.008425 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.986167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42585 (local address 127.0.0.1:38124)
0619 13:14:18.992809 (+ 6642us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:18.992826 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:18.993076 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:18.993487 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:18.993491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:18.994126 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:18.994996 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.995008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.996411 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.996416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.998095 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:18.998112 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.999367 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.999374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:18.999578 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.000146 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:19.000166 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:19.001203 (+ 1037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:19.005585 (+ 4382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:19.005590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:19.005593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:19.006088 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:19.006524 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:19.006528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:19.006532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:19.006669 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:19.007327 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:19.007330 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:19.007492 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 13:14:19.007654 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 13:14:19.007937 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6510,"mutex_wait_us":6,"thread_start_us":6733,"threads_started":1}
I20260619 13:14:19.008713 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:18.985891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.0.97.60:42585)
0619 13:14:18.986064 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:18.986069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:18.992890 (+ 6821us) server_negotiation.cc:407] Connection header received
0619 13:14:18.993205 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:18.993209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:18.993271 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:18.993329 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:18.995124 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.996198 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.998275 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:18.998959 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:18.999057 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.001346 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:19.001351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:19.001355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:19.001387 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:19.005457 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.006202 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.006207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.006210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.006404 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.006796 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.006800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.006803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.007141 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:19.007255 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:19.007807 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 13:14:19.008084 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":59}
I20260619 13:14:19.011265 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 6585
Jun 19 13:14:19 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874856, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 13:14:19.021306 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:19.003474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42585 (local address 127.0.0.1:38126)
0619 13:14:19.003680 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:19.003696 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:19.003789 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:19.006434 (+ 2645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:19.006439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:19.006868 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:19.007040 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:19.007051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.008859 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.008865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:19.010014 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:19.010030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.011025 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.011031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:19.011193 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.011706 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:19.011722 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:19.014605 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:19.018698 (+ 4093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:19.018703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:19.018706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:19.019161 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:19.019554 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:19.019559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:19.019563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:19.019710 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:19.020306 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:19.020311 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:19.020502 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 13:14:19.020662 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 13:14:19.020934 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260619 13:14:19.021631 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:19.003612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38126 (local address 127.0.97.60:42585)
0619 13:14:19.006031 (+ 2419us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:19.006041 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:19.006062 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:14:19.006154 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:19.006160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:19.006240 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:19.007174 (+ 934us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:19.007491 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.008729 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.010413 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.010908 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.011023 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.014827 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:19.014833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:19.014839 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:19.014874 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:19.018568 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.019289 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.019295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.019298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.019444 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.019827 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.019832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.019836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.020188 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:19.020327 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:19.020766 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 13:14:19.020970 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2296,"thread_start_us":2009,"threads_started":1}
Jun 19 13:14:19 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260619 13:14:19.023166 6524 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:42585: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "13c24fa7041b4956a8601b23968dd506"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 42585
}
attrs {
promote: true
}
}
dest_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe"
cas_config_opid_index: 5
I20260619 13:14:19.024161 6524 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: true } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } }
I20260619 13:14:19.026314 7257 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 13c24fa7041b4956a8601b23968dd506 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: true } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } } }
I20260619 13:14:19.030488 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:19.012161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.0.97.1:43431)
0619 13:14:19.013132 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:19.013142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:19.013172 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:14:19.013244 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:19.013255 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:19.013565 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:19.013759 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:19.015141 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.016287 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.017633 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.018238 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.018354 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.023066 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:19.023077 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:19.023091 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:19.023123 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:19.026304 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.027155 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.027165 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.027167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.027307 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.027749 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.027764 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.027768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.028173 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:19.029655 (+ 1482us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:19.029827 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 13:14:19.030110 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":659,"threads_started":1}
I20260619 13:14:19.030997 7131 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 LEARNER]: Refusing update from remote peer a3e798ed377b4da7ab9a1fbdc08cf6fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 13:14:19.031857 6708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:19.031847 7269 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 13:14:19.032536 7212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:19.033464 7257 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:42585
I20260619 13:14:19.034998 7257 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: true } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } } }
I20260619 13:14:19.035810 7257 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:19.044775 7131 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 13c24fa7041b4956a8601b23968dd506 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: true } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } } }
I20260619 13:14:19.049657 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: true } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } } }
I20260619 13:14:19.049989 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: This master's current role is: LEARNER
I20260619 13:14:19.050446 7270 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe: attempting to promote NON_VOTER 5c7b3f0d5a26443f84775a18058284b0 to VOTER
I20260619 13:14:19.051297 7270 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } }
I20260619 13:14:19.054119 7131 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 LEARNER]: Refusing update from remote peer a3e798ed377b4da7ab9a1fbdc08cf6fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 13:14:19.054766 7269 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 13:14:19.057174 7257 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5c7b3f0d5a26443f84775a18058284b0 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } } }
I20260619 13:14:19.057866 7131 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5c7b3f0d5a26443f84775a18058284b0 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } } }
I20260619 13:14:19.061193 7270 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } } }
I20260619 13:14:19.061872 7270 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:19.061690 7257 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } } }
I20260619 13:14:19.062216 7257 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:19.063208 7159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } } }
I20260619 13:14:19.063449 7159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:14:19.066181 7085 master.cc:537] Master@127.0.97.60:42585 shutting down...
I20260619 13:14:19.066319 7212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 1 FOLLOWER]: Refusing update from remote peer a3e798ed377b4da7ab9a1fbdc08cf6fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 13:14:19.067510 7211 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 1 FOLLOWER]: Refusing update from remote peer a3e798ed377b4da7ab9a1fbdc08cf6fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 13:14:19.068723 7257 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 13:14:19.080415 6510 consensus_peers.cc:487] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe -> Peer 13c24fa7041b4956a8601b23968dd506 (127.0.97.60:42585): Couldn't send request to peer 13c24fa7041b4956a8601b23968dd506. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 13:14:19.080713 7085 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:19.081099 7085 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:19.081285 7085 tablet_replica.cc:330] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: stopping tablet replica
I20260619 13:14:19.086683 7085 master.cc:552] Master@127.0.97.60:42585 shutdown complete.
I20260619 13:14:19.086755 7085 master_runner.cc:305] Clearing existing system tablet
I20260619 13:14:19.087076 7085 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:19.087158 7085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:19.089411 7085 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 13:14:19.090692 7276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:19.091001 7085 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:19.091235 7085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 13:14:19.091336 7085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/wal
uuid: "13c24fa7041b4956a8601b23968dd506"
format_stamp: "Formatted at 2026-06-19 13:14:18 on dist-test-slave-mkt3"
server_key: "5ad70227fe37f7807f5f5669b4bab1c5"
server_key_iv: "e9a7b2ef871174840597d410da102fd6"
server_key_version: "encryptionkey@0"
I20260619 13:14:19.094807 7085 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:19.099535 7085 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 13:14:19.099675 7085 log.cc:1196] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 13:14:19.100054 7085 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: Deleting consensus metadata
I20260619 13:14:19.100262 7085 master_runner.cc:315] Copying system tablet from 127.0.97.62:45357
I20260619 13:14:19.102452 7085 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:45357
I20260619 13:14:19.114295 6715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:19.103117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.0.97.62:45357)
0619 13:14:19.103368 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:19.103375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:19.103805 (+ 430us) server_negotiation.cc:407] Connection header received
0619 13:14:19.103973 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:19.103978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:19.104045 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:19.104123 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:19.104920 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.105649 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.106903 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.107436 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.107543 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.109133 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:19.109138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:19.109143 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:19.109173 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:19.111608 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.112181 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.112186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.112190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.112312 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.112947 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.112952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.112956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.113206 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:19.113345 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:19.113703 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 13:14:19.113936 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260619 13:14:19.114324 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:19.103053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:40642)
0619 13:14:19.103694 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:19.103715 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:19.103855 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:19.104162 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:19.104166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:19.104577 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:19.104777 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:19.104786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.105775 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.105781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:19.106751 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:19.106763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.107579 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.107584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:19.107739 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.108305 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:19.108323 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:19.108957 (+ 634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:19.111727 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:19.111732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:19.111737 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:19.112073 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:19.112690 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:19.112693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:19.112695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:19.112842 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:19.113322 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:19.113327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:19.113477 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 13:14:19.113591 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 13:14:19.113903 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":422,"threads_started":1}
I20260619 13:14:19.115103 6544 tablet_copy_service.cc:140] P a3e798ed377b4da7ab9a1fbdc08cf6fe: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 13c24fa7041b4956a8601b23968dd506 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40642)
I20260619 13:14:19.115250 6544 tablet_copy_service.cc:161] P a3e798ed377b4da7ab9a1fbdc08cf6fe: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 13c24fa7041b4956a8601b23968dd506 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40642: session id = 13c24fa7041b4956a8601b23968dd506-00000000000000000000000000000000
I20260619 13:14:19.117832 6544 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 13:14:19.119455 7085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 13:14:19.123920 7085 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: tablet copy: Starting download of 0 data blocks...
I20260619 13:14:19.124172 7085 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: tablet copy: Starting download of 1 WAL segments...
I20260619 13:14:19.126217 7085 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 13:14:19.129057 6544 tablet_copy_service.cc:342] P a3e798ed377b4da7ab9a1fbdc08cf6fe: Request end of tablet copy session 13c24fa7041b4956a8601b23968dd506-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40642
I20260619 13:14:19.129170 6544 tablet_copy_service.cc:433] P a3e798ed377b4da7ab9a1fbdc08cf6fe: ending tablet copy session 13c24fa7041b4956a8601b23968dd506-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 13c24fa7041b4956a8601b23968dd506
I20260619 13:14:19.138550 7085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:19.140793 7283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:19.143066 7284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:19.153498 7286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:19.153960 7085 server_base.cc:1034] running on GCE node
I20260619 13:14:19.154260 7085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:19.154835 7085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:19.156006 7085 hybrid_clock.cc:648] HybridClock initialized: now 1781874859155951 us; error 49 us; skew 500 ppm
I20260619 13:14:19.156206 7085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:19 dist-test-slave-mkt3 krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874859, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:19.157975 7085 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260619 13:14:19.159037 7085 webserver.cc:460] Webserver started at http://127.0.97.60:38789/ using document root <none> and password file <none>
I20260619 13:14:19.159305 7085 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:19.159390 7085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:19.161500 7085 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 13:14:19.162992 7293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:19.163322 7085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 13:14:19.163434 7085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/wal
uuid: "13c24fa7041b4956a8601b23968dd506"
format_stamp: "Formatted at 2026-06-19 13:14:18 on dist-test-slave-mkt3"
server_key: "5ad70227fe37f7807f5f5669b4bab1c5"
server_key_iv: "e9a7b2ef871174840597d410da102fd6"
server_key_version: "encryptionkey@0"
I20260619 13:14:19.163590 7085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:19.180614 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 6585
I20260619 13:14:19.180655 7085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:19.184247 7085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:19.200414 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43431
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45357
--webserver_port=45729
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45357,127.0.97.61:45041
--tserver_master_addrs=127.0.97.62:45357,127.0.97.61:45041,127.0.97.60:42585 with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:14:19.226146 7085 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:42585
I20260619 13:14:19.226161 7345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:42585 every 8 connection(s)
I20260619 13:14:19.227043 7085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/master-2/data/info.pb
I20260619 13:14:19.235401 7346 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:14:19.239627 7346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: Bootstrap starting.
I20260619 13:14:19.253919 7346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:14:19.254336 7346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: Bootstrap complete.
I20260619 13:14:19.255167 7346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } }
I20260619 13:14:19.255385 7346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13c24fa7041b4956a8601b23968dd506, State: Initialized, Role: LEARNER
I20260619 13:14:19.255611 7346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } }
I20260619 13:14:19.256958 7349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } } }
I20260619 13:14:19.257227 7346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:19.257395 7349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [sys.catalog]: This master's current role is: LEARNER
I20260619 13:14:19.273157 7360 catalog_manager.cc:1259] Loaded cluster ID: 260ce7232d764c108a9ed51331674457
I20260619 13:14:19.273247 7360 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: loading cluster ID for follower catalog manager: success
I20260619 13:14:19.275731 7360 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: acquiring CA information for follower catalog manager: success
I20260619 13:14:19.277444 7360 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:14:19.484431 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:19.479136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.0.97.60:42585)
0619 13:14:19.479644 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:19.479648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:19.479664 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:14:19.479720 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:19.479723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:19.479799 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:19.479901 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:19.480303 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.481253 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.482699 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.483472 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.483598 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.483842 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:19.483990 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 13:14:19.484109 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":273,"threads_started":1}
I20260619 13:14:19.484411 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:19.478965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42585 (local address 127.0.0.1:38138)
0619 13:14:19.479124 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:19.479141 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:19.479249 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:19.479953 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:19.479958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:19.479987 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:19.480161 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:19.480170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.481393 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.481397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:19.482557 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:19.482572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.483652 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.483658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:19.483844 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.483877 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:14:19.483960 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 13:14:19.484103 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 13:14:19.485581 7321 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260619 13:14:19.486167 7349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3e798ed377b4da7ab9a1fbdc08cf6fe. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } } }
I20260619 13:14:19.486440 7349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [sys.catalog]: This master's current role is: LEARNER
W20260619 13:14:19.564357 7319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:19.564702 7319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:19.564841 7319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:19.576241 7319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:19.576362 7319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:19.576534 7319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:14:19.587725 7319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45729
--tserver_master_addrs=127.0.97.62:45357,127.0.97.61:45041,127.0.97.60:42585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:19.588977 7319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:19.590287 7319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:19.602516 7366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:19.602548 7367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:19.602474 7369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:19.603300 7319 server_base.cc:1034] running on GCE node
I20260619 13:14:19.603852 7319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:19.604738 7319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:19.605940 7319 hybrid_clock.cc:648] HybridClock initialized: now 1781874859605887 us; error 45 us; skew 500 ppm
I20260619 13:14:19.606253 7319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:19 dist-test-slave-mkt3 krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874859, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:19.612967 7319 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 13:14:19.614316 7319 webserver.cc:460] Webserver started at http://127.0.97.1:45729/ using document root <none> and password file <none>
I20260619 13:14:19.614921 7319 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:19.615041 7319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:19.620333 7319 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 13:14:19.622910 7376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:19.624101 7319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:14:19.624235 7319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "d085b70f4c4d4185be1b6a426e622164"
format_stamp: "Formatted at 2026-06-19 13:14:16 on dist-test-slave-mkt3"
server_key: "bed568b79e0e35ef53aac9b96f77436a"
server_key_iv: "7839f7cf1b878c5d7ea3029f494b61ac"
server_key_version: "encryptionkey@0"
I20260619 13:14:19.625098 7319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:19.648281 7319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:19.651835 7319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:19.652093 7319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:19.653059 7319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:19.654891 7319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:19.654978 7319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:19.655078 7319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:19.655144 7319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:19.686009 7319 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43431
I20260619 13:14:19.686062 7488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43431 every 8 connection(s)
I20260619 13:14:19.687327 7319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:14:19.693635 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 7319
I20260619 13:14:19.694002 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 6719
Jun 19 13:14:19 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874859, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 13:14:19 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874859, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 13:14:19 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874859, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 13:14:19 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260619 13:14:19.719604 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:19.690589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56719 (local address 127.0.97.61:45041)
0619 13:14:19.691105 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:19.691112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:19.694936 (+ 3824us) server_negotiation.cc:407] Connection header received
0619 13:14:19.699439 (+ 4503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:19.699444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:19.699535 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:19.699614 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:19.702384 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.703440 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.706717 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.707317 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.707409 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.712615 (+ 5206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:19.712622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:19.712627 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:19.712667 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:19.715947 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.716845 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.716851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.716855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.716989 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.717559 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.717564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.717567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.717947 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:19.718085 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:19.719026 (+ 941us) server_negotiation.cc:299] Negotiation successful
0619 13:14:19.719235 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":279,"threads_started":1}
I20260619 13:14:19.719830 7361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:19.690022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52005 (local address 127.0.97.60:42585)
0619 13:14:19.690202 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:19.690208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:19.691781 (+ 1573us) server_negotiation.cc:407] Connection header received
0619 13:14:19.700335 (+ 8554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:19.700342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:19.700417 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:19.700484 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:19.702427 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.703447 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.706723 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.707346 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.707425 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.713974 (+ 6549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:19.713980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:19.713985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:19.714024 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:19.716955 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.717785 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.717789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.717792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.717947 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.718339 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.718344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.718347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.718672 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:19.718788 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:19.719283 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 13:14:19.719491 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:14:19.720325 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:19.692331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41499 (local address 127.0.97.62:45357)
0619 13:14:19.692927 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:19.692934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:19.695390 (+ 2456us) server_negotiation.cc:407] Connection header received
0619 13:14:19.699445 (+ 4055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:19.699449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:19.699535 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:19.699660 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:19.702297 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.703344 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.706684 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.707269 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.707364 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.713338 (+ 5974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:19.713345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:19.713350 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:19.713391 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:19.717128 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.717936 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.717941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.717944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.718092 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.718474 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.718479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.718482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.718779 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:19.718886 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:19.719312 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 13:14:19.719496 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":378,"threads_started":1}
I20260619 13:14:19.721457 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:19.694975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.0.97.2:36241)
0619 13:14:19.701250 (+ 6275us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:19.701258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:19.702300 (+ 1042us) server_negotiation.cc:407] Connection header received
0619 13:14:19.702559 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:19.702568 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:19.702793 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:19.702923 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:19.705052 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.706534 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.708252 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.708933 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.709038 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.714324 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:19.714340 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:19.714357 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:19.714398 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:19.718258 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.719069 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.719077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.719080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.719215 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:19.719644 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:19.719649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:19.719652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:19.720050 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:19.720287 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:19.720849 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 13:14:19.721105 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6122,"thread_start_us":5948,"threads_started":1}
I20260619 13:14:19.722184 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:19.690557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45041 (local address 127.0.97.1:56719)
0619 13:14:19.694821 (+ 4264us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:19.694843 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:19.699306 (+ 4463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:19.700136 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:19.700148 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:19.701827 (+ 1679us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:19.702198 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:19.702211 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.703622 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.703627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:19.706564 (+ 2937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:19.706578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.707724 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.707730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:19.708379 (+ 649us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.709325 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:19.709346 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:19.712326 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:19.716119 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:19.716127 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:19.716146 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:19.716715 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:19.717154 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:19.717159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:19.717163 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:19.717426 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:19.718174 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:19.718181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:19.718598 (+ 417us) client_negotiation.cc:769] Sending connection context
0619 13:14:19.719887 (+ 1289us) client_negotiation.cc:240] Negotiation successful
0619 13:14:19.720409 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4071,"mutex_wait_us":269,"thread_start_us":4389,"threads_started":1}
I20260619 13:14:19.722626 6842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:19.722833 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:19.692216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.1:41499)
0619 13:14:19.695296 (+ 3080us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:19.695312 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:19.699050 (+ 3738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:19.700136 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:19.700149 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:19.701081 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:19.702058 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:19.702086 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.703504 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.703509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:19.706541 (+ 3032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:19.706552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.707700 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.707710 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:19.708072 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.709451 (+ 1379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:19.709469 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:19.713085 (+ 3616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:19.717264 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:19.717268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:19.717272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:19.717796 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:19.718223 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:19.718227 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:19.718231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:19.718367 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:19.718945 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:19.718950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:19.719074 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 13:14:19.720542 (+ 1468us) client_negotiation.cc:240] Negotiation successful
0619 13:14:19.722401 (+ 1859us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2981,"mutex_wait_us":1510,"spinlock_wait_cycles":29568,"thread_start_us":1957,"threads_started":1}
I20260619 13:14:19.723242 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:19.690558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42585 (local address 127.0.97.1:52005)
0619 13:14:19.691624 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:19.691667 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:19.699050 (+ 7383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:19.700691 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:19.700695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:19.701609 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:19.702282 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:19.702291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.703632 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.703640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:19.706564 (+ 2924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:19.706578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:19.707705 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:19.707712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:19.708077 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:19.709284 (+ 1207us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:19.709329 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:19.713728 (+ 4399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:19.717104 (+ 3376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:19.717108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:19.717111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:19.717652 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:19.718074 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:19.718077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:19.718081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:19.718226 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:19.718861 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:19.718865 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:19.719011 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 13:14:19.720087 (+ 1076us) client_negotiation.cc:240] Negotiation successful
0619 13:14:19.720734 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":1008,"spinlock_wait_cycles":17664,"thread_start_us":615,"threads_started":1}
I20260619 13:14:19.723722 7490 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45357
I20260619 13:14:19.724057 7490 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:19.724709 7490 heartbeater.cc:511] Master 127.0.97.62:45357 requested a full tablet report, sending...
I20260619 13:14:19.723702 7491 heartbeater.cc:347] Connected to a master server at 127.0.97.61:45041
I20260619 13:14:19.725442 7491 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:19.725775 7491 heartbeater.cc:511] Master 127.0.97.61:45041 requested a full tablet report, sending...
I20260619 13:14:19.726338 7489 heartbeater.cc:347] Connected to a master server at 127.0.97.60:42585
I20260619 13:14:19.726442 7489 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:19.726785 7489 heartbeater.cc:511] Master 127.0.97.60:42585 requested a full tablet report, sending...
I20260619 13:14:19.727576 7121 ts_manager.cc:194] Registered new tserver with Master: d085b70f4c4d4185be1b6a426e622164 (127.0.97.1:43431)
I20260619 13:14:19.727677 6524 ts_manager.cc:194] Re-registered known tserver with Master: d085b70f4c4d4185be1b6a426e622164 (127.0.97.1:43431)
I20260619 13:14:19.728492 7310 ts_manager.cc:194] Registered new tserver with Master: d085b70f4c4d4185be1b6a426e622164 (127.0.97.1:43431)
I20260619 13:14:19.728713 6524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41499
I20260619 13:14:19.809619 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 6719
I20260619 13:14:19.822994 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36241
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45357
--webserver_port=43357
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45357,127.0.97.61:45041
--tserver_master_addrs=127.0.97.62:45357,127.0.97.61:45041,127.0.97.60:42585 with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:14:19.909469 7257 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: true } } }
I20260619 13:14:19.909883 7257 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:19.911875 7502 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe: attempting to promote NON_VOTER 13c24fa7041b4956a8601b23968dd506 to VOTER
I20260619 13:14:19.912420 7502 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } }
I20260619 13:14:19.914074 7321 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 1 LEARNER]: Refusing update from remote peer a3e798ed377b4da7ab9a1fbdc08cf6fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 13:14:19.914084 7131 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 FOLLOWER]: Refusing update from remote peer a3e798ed377b4da7ab9a1fbdc08cf6fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 13:14:19.914579 7257 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 13:14:19.914913 7502 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 13:14:19.917028 7257 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13c24fa7041b4956a8601b23968dd506 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } } }
I20260619 13:14:19.917608 7131 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13c24fa7041b4956a8601b23968dd506 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } } }
I20260619 13:14:19.921048 7257 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } } }
I20260619 13:14:19.921286 7257 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:19.921157 7502 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } } }
I20260619 13:14:19.921454 7502 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:19.921772 7502 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } } }
I20260619 13:14:19.922103 7502 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:19.921661 7321 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 13c24fa7041b4956a8601b23968dd506 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } } }
I20260619 13:14:19.926138 7349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } } }
I20260619 13:14:19.926406 7349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:14:19.927068 7507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } } }
I20260619 13:14:19.927493 7507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 13:14:20.129021 7501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:20.129324 7501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:20.129462 7501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:20.136471 7501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:20.136583 7501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:20.136731 7501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:14:20.146258 7501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43357
--tserver_master_addrs=127.0.97.62:45357,127.0.97.61:45041,127.0.97.60:42585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:20.147503 7501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:20.148804 7501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:20.160990 7512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:20.160956 7513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:20.160912 7515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:20.161732 7501 server_base.cc:1034] running on GCE node
I20260619 13:14:20.162322 7501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:20.163197 7501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:20.164403 7501 hybrid_clock.cc:648] HybridClock initialized: now 1781874860164332 us; error 46 us; skew 500 ppm
I20260619 13:14:20.164729 7501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:20 dist-test-slave-mkt3 krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:20.167708 7501 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 13:14:20.169054 7501 webserver.cc:460] Webserver started at http://127.0.97.2:43357/ using document root <none> and password file <none>
I20260619 13:14:20.169703 7501 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:20.169824 7501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:20.175097 7501 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:14:20.177742 7522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:20.179212 7501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:14:20.179364 7501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "436161bbf1444a99a6611cda7569a9a8"
format_stamp: "Formatted at 2026-06-19 13:14:17 on dist-test-slave-mkt3"
server_key: "a9e2df996b49a933b755ffbb9fbc90d7"
server_key_iv: "55ecfd1cec22c0b0fd6ea372c29b0464"
server_key_version: "encryptionkey@0"
I20260619 13:14:20.180264 7501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:20.191298 7501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:20.195293 7501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:20.195537 7501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:20.196395 7501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:20.197669 7501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:20.197757 7501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:20.197876 7501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:20.197949 7501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260619 13:14:20.226158 7501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36241
I20260619 13:14:20.226150 7634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36241 every 8 connection(s)
I20260619 13:14:20.227535 7501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:14:20.230115 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 7501
I20260619 13:14:20.230376 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 6852
Jun 19 13:14:20 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Jun 19 13:14:20 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 13:14:20 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 13:14:20 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 13:14:20.270920 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.231318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.0.97.3:45933)
0619 13:14:20.244743 (+ 13425us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:20.244754 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:20.244787 (+ 33us) server_negotiation.cc:407] Connection header received
0619 13:14:20.244864 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:20.244874 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:20.245131 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:20.245287 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:20.250443 (+ 5156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.252077 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.254518 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.255355 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.255478 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.261945 (+ 6467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:20.261960 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:20.261978 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:20.262019 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:20.266552 (+ 4533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.268013 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.268021 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.268024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.268156 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.268569 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.268574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.268577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.268965 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:20.269239 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:20.270304 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0619 13:14:20.270545 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13242,"thread_start_us":13079,"threads_started":1}
I20260619 13:14:20.271306 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.247086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44245 (local address 127.0.97.60:42585)
0619 13:14:20.254888 (+ 7802us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:20.254894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:20.254910 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:14:20.254957 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:20.254962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:20.255026 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:20.255117 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:20.256240 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.257348 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.259154 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.259750 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.259859 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.264474 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:20.264480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:20.264485 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:20.264523 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:20.267851 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.268643 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.268648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.268651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.268790 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.269193 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.269198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.269201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.269488 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:20.269627 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:20.270783 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0619 13:14:20.270979 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7633,"thread_start_us":7498,"threads_started":1}
I20260619 13:14:20.272297 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.236423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45041 (local address 127.0.97.2:60233)
0619 13:14:20.250323 (+ 13900us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:20.250342 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:20.250477 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:20.251872 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:20.251885 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:20.253916 (+ 2031us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:20.254949 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.254961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.256299 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.256304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.257988 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.258005 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.259245 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.259251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.259405 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.260263 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:20.260296 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:20.263754 (+ 3458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:20.267192 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:20.267201 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:20.267217 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:20.268320 (+ 1103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:20.268722 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:20.268727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:20.268730 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:20.268948 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:20.269624 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:20.269631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:20.270141 (+ 510us) client_negotiation.cc:769] Sending connection context
0619 13:14:20.270454 (+ 313us) client_negotiation.cc:240] Negotiation successful
0619 13:14:20.270953 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13778,"mutex_wait_us":822,"spinlock_wait_cycles":47232,"thread_start_us":13663,"threads_started":1}
I20260619 13:14:20.272373 6975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:20.272918 7638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.230319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52003 (local address 127.0.97.62:45357)
0619 13:14:20.252512 (+ 22193us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:20.252518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:20.252535 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:14:20.252586 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:20.252591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:20.252659 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:20.254517 (+ 1858us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:20.254660 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.255627 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.258093 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.258707 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.258821 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.264259 (+ 5438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:20.264265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:20.264269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:20.264306 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:20.269098 (+ 4792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.269761 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.269766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.269769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.269921 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.270342 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.270347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.270350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.270647 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:20.270758 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:20.272385 (+ 1627us) server_negotiation.cc:299] Negotiation successful
0619 13:14:20.272587 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22047,"thread_start_us":22426,"threads_started":1}
I20260619 13:14:20.273305 7646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.247106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42585 (local address 127.0.97.2:44245)
0619 13:14:20.254211 (+ 7105us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:20.254237 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:20.254333 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:20.255284 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:20.255289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:20.255733 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:20.256061 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.256071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.257500 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.257505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.259010 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.259021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.260046 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.260053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.260247 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.260764 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:20.260790 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:20.264274 (+ 3484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:20.267991 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:20.267995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:20.267998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:20.268526 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:20.268911 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:20.268916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:20.268920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:20.269082 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:20.269683 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:20.269687 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:20.270142 (+ 455us) client_negotiation.cc:769] Sending connection context
0619 13:14:20.270454 (+ 312us) client_negotiation.cc:240] Negotiation successful
0619 13:14:20.273031 (+ 2577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6986,"mutex_wait_us":2243,"spinlock_wait_cycles":7296,"thread_start_us":6841,"threads_started":1}
I20260619 13:14:20.274226 7643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.236889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60233 (local address 127.0.97.61:45041)
0619 13:14:20.249363 (+ 12474us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:20.249370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:20.250583 (+ 1213us) server_negotiation.cc:407] Connection header received
0619 13:14:20.250653 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:20.250659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:20.250742 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:20.250834 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:20.255128 (+ 4294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.256138 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.258236 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.258865 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.258985 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.263999 (+ 5014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:20.264005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:20.264009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:20.264051 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:20.267065 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.268462 (+ 1397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.268468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.268472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.268598 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.269084 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.269089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.269092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.269429 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:20.269559 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:20.273666 (+ 4107us) server_negotiation.cc:299] Negotiation successful
0619 13:14:20.273909 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12336,"thread_start_us":12208,"threads_started":1}
I20260619 13:14:20.274511 7635 heartbeater.cc:347] Connected to a master server at 127.0.97.60:42585
I20260619 13:14:20.274847 7635 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:20.274967 7640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.231044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.2:52003)
0619 13:14:20.244046 (+ 13002us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:20.244102 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:20.247971 (+ 3869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:20.252913 (+ 4942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:20.252918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:20.253433 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:20.254334 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.254356 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.255815 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.255824 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.257923 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.257943 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.259084 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.259091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.259352 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.260260 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:20.260296 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:20.264020 (+ 3724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:20.269225 (+ 5205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:20.269230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:20.269233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:20.269640 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:20.270043 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:20.270047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:20.270050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:20.270197 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:20.270822 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:20.270826 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:20.270968 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 13:14:20.271083 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 13:14:20.271330 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12701,"mutex_wait_us":8,"spinlock_wait_cycles":3072,"thread_start_us":14709,"threads_started":1}
I20260619 13:14:20.275588 7635 heartbeater.cc:511] Master 127.0.97.60:42585 requested a full tablet report, sending...
I20260619 13:14:20.276180 7637 heartbeater.cc:347] Connected to a master server at 127.0.97.61:45041
I20260619 13:14:20.276279 7637 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:20.276573 7637 heartbeater.cc:511] Master 127.0.97.61:45041 requested a full tablet report, sending...
I20260619 13:14:20.277565 7636 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45357
I20260619 13:14:20.277684 7636 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:20.277765 7121 ts_manager.cc:194] Registered new tserver with Master: 436161bbf1444a99a6611cda7569a9a8 (127.0.97.2:36241)
I20260619 13:14:20.277822 7310 ts_manager.cc:194] Registered new tserver with Master: 436161bbf1444a99a6611cda7569a9a8 (127.0.97.2:36241)
I20260619 13:14:20.278080 7636 heartbeater.cc:511] Master 127.0.97.62:45357 requested a full tablet report, sending...
I20260619 13:14:20.278883 6524 ts_manager.cc:194] Re-registered known tserver with Master: 436161bbf1444a99a6611cda7569a9a8 (127.0.97.2:36241)
I20260619 13:14:20.279610 6524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52003
I20260619 13:14:20.361239 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 6852
I20260619 13:14:20.373612 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45933
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45357
--webserver_port=41725
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45357,127.0.97.61:45041
--tserver_master_addrs=127.0.97.62:45357,127.0.97.61:45041,127.0.97.60:42585 with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:20.680014 7648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:20.680281 7648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:20.680392 7648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:20.687661 7648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:20.687762 7648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:20.687897 7648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:14:20.697623 7648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41725
--tserver_master_addrs=127.0.97.62:45357,127.0.97.61:45041,127.0.97.60:42585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:20.698921 7648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:20.700178 7648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:20.712245 7653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:20.712152 7654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:20.712219 7656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:20.713008 7648 server_base.cc:1034] running on GCE node
I20260619 13:14:20.713562 7648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:20.714435 7648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:20.715615 7648 hybrid_clock.cc:648] HybridClock initialized: now 1781874860715569 us; error 41 us; skew 500 ppm
I20260619 13:14:20.715902 7648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:20 dist-test-slave-mkt3 krb5kdc[6479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:20.718742 7648 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 13:14:20.720052 7648 webserver.cc:460] Webserver started at http://127.0.97.3:41725/ using document root <none> and password file <none>
I20260619 13:14:20.720664 7648 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:20.720778 7648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:20.725713 7648 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:14:20.728207 7663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:20.729650 7648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:14:20.729800 7648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "410babf59db6434c9d72b772ac83f5f9"
format_stamp: "Formatted at 2026-06-19 13:14:17 on dist-test-slave-mkt3"
server_key: "35d6c803bdd6ea0d513f6606840b6f06"
server_key_iv: "290766c59bdd787ce82825eb6bad80a8"
server_key_version: "encryptionkey@0"
I20260619 13:14:20.730937 7648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:20.731148 7490 heartbeater.cc:503] Master 127.0.97.62:45357 was elected leader, sending a full tablet report...
I20260619 13:14:20.748011 7648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:20.751626 7648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:20.751830 7648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:20.752732 7648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:20.754004 7648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:20.754082 7648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:20.754164 7648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:20.754215 7648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:20.783408 7648 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45933
I20260619 13:14:20.783494 7775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45933 every 8 connection(s)
I20260619 13:14:20.784775 7648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:14:20.790684 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 7648
Jun 19 13:14:20 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 13:14:20 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 13:14:20 dist-test-slave-mkt3 krb5kdc[6479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 13:14:20.817463 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.787389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36679 (local address 127.0.97.61:45041)
0619 13:14:20.793068 (+ 5679us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:20.793074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:20.795319 (+ 2245us) server_negotiation.cc:407] Connection header received
0619 13:14:20.796900 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:20.796908 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:20.797009 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:20.797086 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:20.800090 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.801098 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.803549 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.804159 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.804248 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.809387 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:20.809392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:20.809397 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:20.809439 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:20.812969 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.814329 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.814335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.814339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.814502 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.815485 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.815489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.815493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.815813 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:20.815964 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:20.816899 (+ 935us) server_negotiation.cc:299] Negotiation successful
0619 13:14:20.817097 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5542,"thread_start_us":5448,"threads_started":1}
I20260619 13:14:20.818709 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.794616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.3:48431)
0619 13:14:20.797278 (+ 2662us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:20.797295 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:20.797390 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:20.798515 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:20.798521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:20.799239 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:20.800080 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.800092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.801618 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.801624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.803430 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.803445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.804504 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.804511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.804957 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.805909 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:20.805949 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:20.808715 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:20.814619 (+ 5904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:20.814623 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:20.814626 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:20.815074 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:20.816011 (+ 937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:20.816015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:20.816018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:20.816218 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:20.817800 (+ 1582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:20.817806 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:20.817967 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 13:14:20.818114 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 13:14:20.818354 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2552,"mutex_wait_us":524,"thread_start_us":2645,"threads_started":1}
I20260619 13:14:20.819888 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.794710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48431 (local address 127.0.97.62:45357)
0619 13:14:20.798156 (+ 3446us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:20.798163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:20.798183 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:14:20.798242 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:20.798248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:20.798323 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:20.799378 (+ 1055us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:20.800254 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.801437 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.803628 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.804232 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.804753 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.808907 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:20.808915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:20.808920 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:20.808963 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:20.814167 (+ 5204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.815198 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.815203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.815205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.815355 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.816351 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.816356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.816359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.816648 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:20.816758 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:20.819353 (+ 2595us) server_negotiation.cc:299] Negotiation successful
0619 13:14:20.819552 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3321,"thread_start_us":3228,"threads_started":1}
I20260619 13:14:20.821120 7777 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45357
I20260619 13:14:20.821447 7777 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:20.822342 7777 heartbeater.cc:511] Master 127.0.97.62:45357 requested a full tablet report, sending...
I20260619 13:14:20.823918 6524 ts_manager.cc:194] Re-registered known tserver with Master: 410babf59db6434c9d72b772ac83f5f9 (127.0.97.3:45933)
I20260619 13:14:20.824997 6524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48431
I20260619 13:14:20.828163 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.788484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45041 (local address 127.0.97.3:36679)
0619 13:14:20.795210 (+ 6726us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:20.795232 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:20.796577 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:20.797550 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:20.797563 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:20.798520 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:20.799924 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.799935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.801279 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.801284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.803399 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.803413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.804548 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.804553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.804960 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.805976 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:20.805994 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:20.809207 (+ 3213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:20.813092 (+ 3885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:20.813101 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:20.813117 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:20.813644 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:20.814592 (+ 948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:20.814597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:20.814600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:20.814836 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:20.815923 (+ 1087us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:20.815931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:20.816337 (+ 406us) client_negotiation.cc:769] Sending connection context
0619 13:14:20.816640 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 13:14:20.817119 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6590,"mutex_wait_us":664,"thread_start_us":8575,"threads_started":1}
I20260619 13:14:20.828943 7778 heartbeater.cc:347] Connected to a master server at 127.0.97.61:45041
I20260619 13:14:20.829107 7778 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:20.829986 7778 heartbeater.cc:511] Master 127.0.97.61:45041 requested a full tablet report, sending...
I20260619 13:14:20.831080 7121 ts_manager.cc:194] Registered new tserver with Master: 410babf59db6434c9d72b772ac83f5f9 (127.0.97.3:45933)
I20260619 13:14:20.839200 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.807911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.0.97.62:45357)
0619 13:14:20.815479 (+ 7568us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:20.815484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:20.815497 (+ 13us) server_negotiation.cc:407] Connection header received
0619 13:14:20.815543 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:20.815548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:20.815612 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:20.815700 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:20.823572 (+ 7872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.824781 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.827362 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.828029 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.828144 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.832647 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:20.832654 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:20.832658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:20.832693 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:20.835837 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.836598 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.836604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.836608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.836753 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.837175 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.837180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.837183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.837493 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:20.837689 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:20.838653 (+ 964us) server_negotiation.cc:299] Negotiation successful
0619 13:14:20.838857 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5320,"thread_start_us":5183,"threads_started":1}
I20260619 13:14:20.840252 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.787961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42585 (local address 127.0.97.3:38733)
0619 13:14:20.793623 (+ 5662us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:20.793675 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:20.796576 (+ 2901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:20.798345 (+ 1769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:20.798349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:20.798831 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:20.799563 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.799581 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.801005 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.801010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.803420 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.803428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.804529 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.804536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.805437 (+ 901us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.809393 (+ 3956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:20.809413 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:20.831625 (+ 22212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:20.837024 (+ 5399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:20.837029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:20.837032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:20.837466 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:20.837892 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:20.837897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:20.837900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:20.838047 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:20.838671 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:20.838676 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:20.838808 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 13:14:20.838928 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 13:14:20.839211 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5373,"mutex_wait_us":416,"thread_start_us":5168,"threads_started":1}
I20260619 13:14:20.841190 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.787376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38733 (local address 127.0.97.60:42585)
0619 13:14:20.794167 (+ 6791us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:20.794173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:20.794194 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:14:20.798094 (+ 3900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:20.798099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:20.798173 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:20.799294 (+ 1121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:20.799715 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.800849 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.803550 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.804157 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.804288 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.831853 (+ 27565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:20.831860 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:20.831865 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:20.831908 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:20.836668 (+ 4760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.837610 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.837615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.837618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.837744 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.838187 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.838192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.838195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.838530 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:20.840549 (+ 2019us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:20.840666 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 13:14:20.840850 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6697,"thread_start_us":6583,"threads_started":1}
I20260619 13:14:20.841959 7776 heartbeater.cc:347] Connected to a master server at 127.0.97.60:42585
I20260619 13:14:20.842087 7776 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:20.842394 7776 heartbeater.cc:511] Master 127.0.97.60:42585 requested a full tablet report, sending...
I20260619 13:14:20.845443 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.813470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.0.97.61:45041)
0619 13:14:20.817742 (+ 4272us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:20.817747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:20.828069 (+ 10322us) server_negotiation.cc:407] Connection header received
0619 13:14:20.828950 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:20.828955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:20.829035 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:20.829126 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:20.831851 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.832826 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.834967 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.836056 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.836175 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.839240 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:20.839246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:20.839250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:20.839285 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:20.842845 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.843568 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.843573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.843577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.843702 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.844030 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.844035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.844039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.844290 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:20.844511 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:20.844806 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 13:14:20.845112 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4178,"thread_start_us":6767,"threads_started":1}
I20260619 13:14:20.846531 7310 ts_manager.cc:194] Registered new tserver with Master: 410babf59db6434c9d72b772ac83f5f9 (127.0.97.3:45933)
I20260619 13:14:20.848623 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.815386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.0.97.60:42585)
0619 13:14:20.832331 (+ 16945us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:20.832337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:20.832713 (+ 376us) server_negotiation.cc:407] Connection header received
0619 13:14:20.832764 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:20.832769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:20.832858 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:20.832990 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:20.834890 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.836360 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.838149 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.838806 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.838921 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.843168 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:20.843174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:20.843178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:20.843205 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:20.846187 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.846951 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.846956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.846960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.847090 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:20.847405 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:20.847410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:20.847413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:20.847713 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:20.847934 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:20.848121 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 13:14:20.848308 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16788,"thread_start_us":16957,"threads_started":1}
I20260619 13:14:20.843309 6524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:14:20.851984 6524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:14:20.881781 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.869979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37240 (local address 127.0.97.1:43431)
0619 13:14:20.870807 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:20.870816 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:20.874112 (+ 3296us) server_negotiation.cc:407] Connection header received
0619 13:14:20.874365 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:20.874374 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:20.874646 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:20.874844 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:20.875648 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.876765 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.879053 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.880741 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.880844 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.881078 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:20.881357 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 13:14:20.881509 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":610,"threads_started":1}
I20260619 13:14:20.881886 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.869461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43431 (local address 127.0.0.1:37240)
0619 13:14:20.874002 (+ 4541us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:20.874022 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:20.874203 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:20.875046 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:20.875051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:20.875073 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:20.875450 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.875462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.876898 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.876901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.878911 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.878925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.880942 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.880948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.881126 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.881219 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 13:14:20.881296 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 13:14:20.881427 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":36,"thread_start_us":1339,"threads_started":1}
I20260619 13:14:20.882134 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.870449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36241 (local address 127.0.0.1:43978)
0619 13:14:20.871255 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:20.871274 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:20.871389 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:20.875271 (+ 3882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:20.875275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:20.875298 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:20.875712 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.875725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.877424 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.877429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.879690 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.879707 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.880981 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.880987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.881229 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.881261 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 13:14:20.881343 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 13:14:20.881475 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":67,"thread_start_us":1146,"threads_started":1}
I20260619 13:14:20.882782 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.870692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.0.97.2:36241)
0619 13:14:20.874628 (+ 3936us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:20.874636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:20.874663 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:14:20.874752 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:20.874759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:20.875055 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:20.875290 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:20.875888 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.877310 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.879865 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.880770 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.880890 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.881095 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:20.882354 (+ 1259us) server_negotiation.cc:299] Negotiation successful
0619 13:14:20.882523 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3792,"thread_start_us":3690,"threads_started":1}
I20260619 13:14:20.885267 7424 tablet_service.cc:1467] Processing CreateTablet for tablet 8c0891b77a654ff3bed4dda536bcf5f1 (DEFAULT_TABLE table=test-table [id=eb33d9ffbe994f209b6bfdb62ef6a344]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:20.885773 7570 tablet_service.cc:1467] Processing CreateTablet for tablet 8c0891b77a654ff3bed4dda536bcf5f1 (DEFAULT_TABLE table=test-table [id=eb33d9ffbe994f209b6bfdb62ef6a344]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:20.886444 7424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c0891b77a654ff3bed4dda536bcf5f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:20.887131 7570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c0891b77a654ff3bed4dda536bcf5f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:20.887144 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.869020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45933 (local address 127.0.0.1:47402)
0619 13:14:20.869550 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:20.869584 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:20.870167 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:20.875576 (+ 5409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:20.875580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:20.875607 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:20.875982 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.875992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.877719 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.877724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.879694 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.879707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.886338 (+ 6631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.886345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.886529 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.886560 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:14:20.886668 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 13:14:20.886799 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":64,"thread_start_us":1118,"threads_started":1}
I20260619 13:14:20.887187 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.869389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.0.97.3:45933)
0619 13:14:20.874851 (+ 5462us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:20.874898 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:20.874933 (+ 35us) server_negotiation.cc:407] Connection header received
0619 13:14:20.875023 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:20.875046 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:20.875392 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:20.875594 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:20.876162 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.877603 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.881164 (+ 3561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.882781 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.882907 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.883096 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:20.886734 (+ 3638us) server_negotiation.cc:299] Negotiation successful
0619 13:14:20.886896 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5301,"thread_start_us":4630,"threads_started":1}
I20260619 13:14:20.890736 7711 tablet_service.cc:1467] Processing CreateTablet for tablet 8c0891b77a654ff3bed4dda536bcf5f1 (DEFAULT_TABLE table=test-table [id=eb33d9ffbe994f209b6bfdb62ef6a344]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:20.892067 7711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c0891b77a654ff3bed4dda536bcf5f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:20.898754 7810 tablet_bootstrap.cc:492] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164: Bootstrap starting.
I20260619 13:14:20.901773 7810 tablet_bootstrap.cc:654] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:20.902138 7811 tablet_bootstrap.cc:492] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8: Bootstrap starting.
I20260619 13:14:20.903113 7810 log.cc:824] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:20.905161 7811 tablet_bootstrap.cc:654] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:20.906313 7813 tablet_bootstrap.cc:492] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9: Bootstrap starting.
I20260619 13:14:20.906458 7810 tablet_bootstrap.cc:492] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164: No bootstrap required, opened a new log
I20260619 13:14:20.906795 7810 ts_tablet_manager.cc:1397] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:14:20.907109 7811 log.cc:824] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:20.909960 7813 tablet_bootstrap.cc:654] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:20.910000 7811 tablet_bootstrap.cc:492] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8: No bootstrap required, opened a new log
I20260619 13:14:20.910233 7811 ts_tablet_manager.cc:1397] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260619 13:14:20.910929 7813 log.cc:824] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:20.913112 7813 tablet_bootstrap.cc:492] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9: No bootstrap required, opened a new log
I20260619 13:14:20.913319 7813 ts_tablet_manager.cc:1397] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260619 13:14:20.916822 7810 raft_consensus.cc:348] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } } peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } }
I20260619 13:14:20.917222 7810 raft_consensus.cc:374] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:20.917321 7810 raft_consensus.cc:729] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d085b70f4c4d4185be1b6a426e622164, State: Initialized, Role: FOLLOWER
I20260619 13:14:20.917896 7810 consensus_queue.cc:260] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } } peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } }
I20260619 13:14:20.919580 7810 ts_tablet_manager.cc:1428] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164: Time spent starting tablet: real 0.013s user 0.000s sys 0.012s
I20260619 13:14:20.922243 7811 raft_consensus.cc:348] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } } peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } }
I20260619 13:14:20.922664 7811 raft_consensus.cc:374] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:20.922775 7811 raft_consensus.cc:729] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436161bbf1444a99a6611cda7569a9a8, State: Initialized, Role: FOLLOWER
I20260619 13:14:20.923015 7813 raft_consensus.cc:348] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } } peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } }
I20260619 13:14:20.923409 7813 raft_consensus.cc:374] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:20.923501 7813 raft_consensus.cc:729] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410babf59db6434c9d72b772ac83f5f9, State: Initialized, Role: FOLLOWER
I20260619 13:14:20.923439 7811 consensus_queue.cc:260] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } } peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } }
I20260619 13:14:20.924046 7813 consensus_queue.cc:260] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } } peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } }
I20260619 13:14:20.926347 7811 ts_tablet_manager.cc:1428] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260619 13:14:20.926529 7813 ts_tablet_manager.cc:1428] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260619 13:14:20.926584 7636 heartbeater.cc:503] Master 127.0.97.62:45357 was elected leader, sending a full tablet report...
I20260619 13:14:20.926918 7777 heartbeater.cc:503] Master 127.0.97.62:45357 was elected leader, sending a full tablet report...
W20260619 13:14:20.940667 7492 tablet.cc:2369] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:14:20.984362 7817 raft_consensus.cc:482] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:14:20.984633 7817 raft_consensus.cc:504] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } } peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } }
I20260619 13:14:20.986223 7817 leader_election.cc:288] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 410babf59db6434c9d72b772ac83f5f9 (127.0.97.3:45933), d085b70f4c4d4185be1b6a426e622164 (127.0.97.1:43431)
I20260619 13:14:20.993220 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.986518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42335 (local address 127.0.97.3:45933)
0619 13:14:20.986771 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:20.986776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:20.987866 (+ 1090us) server_negotiation.cc:407] Connection header received
0619 13:14:20.988046 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:20.988050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:20.988104 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:20.988174 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:20.988736 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.989431 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.991375 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.992468 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.992595 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.992694 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:20.992941 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 13:14:20.993034 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260619 13:14:20.993413 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.986440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45933 (local address 127.0.97.2:42335)
0619 13:14:20.987764 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:20.987779 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:20.987902 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:20.988211 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:20.988216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:20.988238 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:20.988589 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.988598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.989553 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.989558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.991244 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.991255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.992651 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.992655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.992780 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.992813 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:14:20.992920 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 13:14:20.993041 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1214,"mutex_wait_us":39,"thread_start_us":924,"threads_started":1}
I20260619 13:14:20.993538 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.986694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47375 (local address 127.0.97.1:43431)
0619 13:14:20.986873 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:20.986879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:20.987953 (+ 1074us) server_negotiation.cc:407] Connection header received
0619 13:14:20.988124 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:20.988128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:20.988195 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:20.988267 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:20.988792 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.989598 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.991589 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.992725 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.992826 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.992899 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:20.993258 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 13:14:20.993379 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:14:20.993705 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:20.986469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43431 (local address 127.0.97.2:47375)
0619 13:14:20.987842 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:20.987859 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:20.987995 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:20.988295 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:20.988299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:20.988325 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:20.988655 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.988664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.989753 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.989757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.991450 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:20.991461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:20.992938 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:20.992943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:20.993093 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:20.993123 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 13:14:20.993211 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 13:14:20.993336 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1283,"mutex_wait_us":37,"thread_start_us":1009,"threads_started":1}
I20260619 13:14:20.994349 7731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0891b77a654ff3bed4dda536bcf5f1" candidate_uuid: "436161bbf1444a99a6611cda7569a9a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410babf59db6434c9d72b772ac83f5f9" is_pre_election: true
I20260619 13:14:20.994544 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0891b77a654ff3bed4dda536bcf5f1" candidate_uuid: "436161bbf1444a99a6611cda7569a9a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d085b70f4c4d4185be1b6a426e622164" is_pre_election: true
I20260619 13:14:20.994781 7731 raft_consensus.cc:2454] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 436161bbf1444a99a6611cda7569a9a8 in term 0.
I20260619 13:14:20.994988 7444 raft_consensus.cc:2454] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 436161bbf1444a99a6611cda7569a9a8 in term 0.
I20260619 13:14:20.995366 7525 leader_election.cc:302] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 410babf59db6434c9d72b772ac83f5f9, 436161bbf1444a99a6611cda7569a9a8; no voters:
I20260619 13:14:20.995759 7817 raft_consensus.cc:2781] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260619 13:14:20.995776 7644 tablet.cc:2369] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:14:20.995932 7817 raft_consensus.cc:482] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:14:20.996054 7817 raft_consensus.cc:3037] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:20.998267 7817 raft_consensus.cc:504] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } } peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } }
I20260619 13:14:20.998813 7817 leader_election.cc:288] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [CANDIDATE]: Term 1 election: Requested vote from peers 410babf59db6434c9d72b772ac83f5f9 (127.0.97.3:45933), d085b70f4c4d4185be1b6a426e622164 (127.0.97.1:43431)
I20260619 13:14:20.999189 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0891b77a654ff3bed4dda536bcf5f1" candidate_uuid: "436161bbf1444a99a6611cda7569a9a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d085b70f4c4d4185be1b6a426e622164"
I20260619 13:14:20.999634 7444 raft_consensus.cc:3037] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:20.999614 7731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0891b77a654ff3bed4dda536bcf5f1" candidate_uuid: "436161bbf1444a99a6611cda7569a9a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410babf59db6434c9d72b772ac83f5f9"
I20260619 13:14:20.999811 7731 raft_consensus.cc:3037] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:21.001878 7731 raft_consensus.cc:2454] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436161bbf1444a99a6611cda7569a9a8 in term 1.
I20260619 13:14:21.001899 7444 raft_consensus.cc:2454] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436161bbf1444a99a6611cda7569a9a8 in term 1.
I20260619 13:14:21.002246 7526 leader_election.cc:302] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 436161bbf1444a99a6611cda7569a9a8, d085b70f4c4d4185be1b6a426e622164; no voters:
I20260619 13:14:21.002525 7817 raft_consensus.cc:2781] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:21.002910 7817 raft_consensus.cc:686] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [term 1 LEADER]: Becoming Leader. State: Replica: 436161bbf1444a99a6611cda7569a9a8, State: Running, Role: LEADER
I20260619 13:14:21.003376 7817 consensus_queue.cc:237] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } } peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } }
I20260619 13:14:21.008059 6523 catalog_manager.cc:5495] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 436161bbf1444a99a6611cda7569a9a8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "436161bbf1444a99a6611cda7569a9a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } health_report { overall_health: HEALTHY } } }
I20260619 13:14:21.031072 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:21.026156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.0.97.2:36241)
0619 13:14:21.026382 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:21.026387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:21.026403 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:14:21.026482 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:21.026485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:21.026557 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:21.026690 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:14:21.027054 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:21.027746 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:21.029032 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:21.029468 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:21.029558 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:21.030057 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:14:21.030557 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:14:21.030665 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:21.030767 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 13:14:21.030827 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:14:21.038904 7731 raft_consensus.cc:1261] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9 [term 1 FOLLOWER]: Refusing update from remote peer 436161bbf1444a99a6611cda7569a9a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:21.039688 7444 raft_consensus.cc:1261] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164 [term 1 FOLLOWER]: Refusing update from remote peer 436161bbf1444a99a6611cda7569a9a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:21.039783 7821 consensus_queue.cc:1035] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:21.040468 7821 consensus_queue.cc:1035] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 13:14:21.042743 7779 tablet.cc:2369] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:14:21.057384 7824 mvcc.cc:205] Tried to move back new op lower bound from 7298559430806298624 to 7298559430673461248. Current Snapshot: MvccSnapshot[applied={T|T < 7298559430806298624 or (T in {7298559430806298624})}]
I20260619 13:14:21.066203 6523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:21.066440 6523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:21.071731 6523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3e798ed377b4da7ab9a1fbdc08cf6fe: Sending DeleteTablet for 3 replicas of tablet 8c0891b77a654ff3bed4dda536bcf5f1
I20260619 13:14:21.072779 7711 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c0891b77a654ff3bed4dda536bcf5f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47402
I20260619 13:14:21.073432 7570 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c0891b77a654ff3bed4dda536bcf5f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43978
I20260619 13:14:21.073576 7833 tablet_replica.cc:330] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9: stopping tablet replica
I20260619 13:14:21.073936 7833 raft_consensus.cc:2229] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:21.074280 7834 tablet_replica.cc:330] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8: stopping tablet replica
I20260619 13:14:21.074393 7833 raft_consensus.cc:2258] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:21.074703 7834 raft_consensus.cc:2229] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:14:21.075165 7834 raft_consensus.cc:2258] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:21.075116 7424 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c0891b77a654ff3bed4dda536bcf5f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37240
I20260619 13:14:21.075819 7835 tablet_replica.cc:330] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164: stopping tablet replica
I20260619 13:14:21.076155 7835 raft_consensus.cc:2229] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:21.076594 7835 raft_consensus.cc:2258] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:21.077802 7833 ts_tablet_manager.cc:1905] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:21.079284 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 6492
I20260619 13:14:21.080193 6554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:21.080751 7834 ts_tablet_manager.cc:1905] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:21.085597 7833 ts_tablet_manager.cc:1918] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:21.085873 7833 log.cc:1196] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/8c0891b77a654ff3bed4dda536bcf5f1
I20260619 13:14:21.086436 7833 ts_tablet_manager.cc:1939] T 8c0891b77a654ff3bed4dda536bcf5f1 P 410babf59db6434c9d72b772ac83f5f9: Deleting consensus metadata
I20260619 13:14:21.087252 7834 ts_tablet_manager.cc:1918] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:21.087405 7834 log.cc:1196] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/8c0891b77a654ff3bed4dda536bcf5f1
I20260619 13:14:21.087970 7834 ts_tablet_manager.cc:1939] T 8c0891b77a654ff3bed4dda536bcf5f1 P 436161bbf1444a99a6611cda7569a9a8: Deleting consensus metadata
I20260619 13:14:21.088353 6510 catalog_manager.cc:4841] TS 410babf59db6434c9d72b772ac83f5f9 (127.0.97.3:45933): tablet 8c0891b77a654ff3bed4dda536bcf5f1 (table test-table [id=eb33d9ffbe994f209b6bfdb62ef6a344]) successfully deleted
I20260619 13:14:21.091473 7835 ts_tablet_manager.cc:1905] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:21.096105 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:21.077350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.0.97.61:45041)
0619 13:14:21.077505 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:21.077510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:21.078017 (+ 507us) server_negotiation.cc:407] Connection header received
0619 13:14:21.078267 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:21.078272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:21.078345 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:21.078409 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:21.082223 (+ 3814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:21.083272 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:21.085105 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:21.085788 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:21.085900 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:21.089154 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:21.089161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:21.089166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:21.089202 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:21.092903 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:21.093700 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:21.093705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:21.093709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:21.093897 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:21.094391 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:21.094396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:21.094400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:21.094726 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:21.094939 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:21.095554 (+ 615us) server_negotiation.cc:299] Negotiation successful
0619 13:14:21.095773 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:14:21.097931 7835 ts_tablet_manager.cc:1918] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:21.098078 7835 log.cc:1196] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/8c0891b77a654ff3bed4dda536bcf5f1
I20260619 13:14:21.098619 7835 ts_tablet_manager.cc:1939] T 8c0891b77a654ff3bed4dda536bcf5f1 P d085b70f4c4d4185be1b6a426e622164: Deleting consensus metadata
I20260619 13:14:21.110245 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:21.079093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.0.97.60:42585)
0619 13:14:21.079796 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:21.079802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:21.098253 (+ 18451us) server_negotiation.cc:407] Connection header received
0619 13:14:21.098518 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:21.098524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:21.098609 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:21.098682 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:21.100420 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:21.101115 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:21.102382 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:21.102801 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:21.102910 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:21.105410 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:21.105415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:21.105418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:21.105449 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:21.107975 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:21.108517 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:21.108522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:21.108525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:21.108635 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:21.108952 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:21.108956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:21.108960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:21.109201 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:21.109434 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:21.109716 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 13:14:21.109949 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565}
I20260619 13:14:21.221648 6510 catalog_manager.cc:4841] TS 436161bbf1444a99a6611cda7569a9a8 (127.0.97.2:36241): tablet 8c0891b77a654ff3bed4dda536bcf5f1 (table test-table [id=eb33d9ffbe994f209b6bfdb62ef6a344]) successfully deleted
W20260619 13:14:21.221918 6510 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.2:36241, user_credentials={real_user=slave}} blocked reactor thread for 132354us
I20260619 13:14:21.222235 6511 catalog_manager.cc:4841] TS d085b70f4c4d4185be1b6a426e622164 (127.0.97.1:43431): tablet 8c0891b77a654ff3bed4dda536bcf5f1 (table test-table [id=eb33d9ffbe994f209b6bfdb62ef6a344]) successfully deleted
I20260619 13:14:21.222254 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 6492
I20260619 13:14:21.274133 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:21.255838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.0.97.61:45041)
0619 13:14:21.256060 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:21.256067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:21.259614 (+ 3547us) server_negotiation.cc:407] Connection header received
0619 13:14:21.259877 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:21.259883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:21.259964 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:21.260031 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:21.262358 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:21.263293 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:21.264749 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:21.265317 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:21.265430 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:21.268717 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:21.268723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:21.268727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:21.268764 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:21.271633 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:21.272283 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:21.272288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:21.272292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:21.272415 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:21.272750 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:21.272754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:21.272757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:21.273061 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:21.273270 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:21.273617 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 13:14:21.273816 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:14:21.274292 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:21.259239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38156 (local address 127.0.97.60:42585)
0619 13:14:21.259407 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:21.259413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:21.260183 (+ 770us) server_negotiation.cc:407] Connection header received
0619 13:14:21.260311 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:21.260316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:21.260393 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:21.260472 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:21.261978 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:21.262972 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:21.264978 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:21.265565 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:21.265683 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:21.268198 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:21.268203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:21.268206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:21.268237 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:21.271229 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:21.271923 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:21.271929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:21.271932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:21.272057 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:21.272423 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:21.272427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:21.272430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:21.272730 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:21.272933 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:21.273719 (+ 786us) server_negotiation.cc:299] Negotiation successful
0619 13:14:21.273941 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 13:14:22.056066 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:22.055235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.1:58573)
0619 13:14:22.055710 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:22.055857 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":232,"threads_started":1}
W20260619 13:14:22.056439 7490 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45357 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
I20260619 13:14:22.059909 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:22.059211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.2:52315)
0619 13:14:22.059644 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:22.059760 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":213,"threads_started":1}
W20260619 13:14:22.060269 7636 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45357 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
I20260619 13:14:22.062700 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:22.062090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.3:33041)
0619 13:14:22.062432 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:22.062534 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":166,"threads_started":1}
W20260619 13:14:22.063016 7777 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45357 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
I20260619 13:14:22.572088 7847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a3e798ed377b4da7ab9a1fbdc08cf6fe)
I20260619 13:14:22.572352 7847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } }
I20260619 13:14:22.573261 7848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a3e798ed377b4da7ab9a1fbdc08cf6fe)
I20260619 13:14:22.573493 7848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } }
I20260619 13:14:22.573872 7847 leader_election.cc:288] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3e798ed377b4da7ab9a1fbdc08cf6fe (127.0.97.62:45357), 5c7b3f0d5a26443f84775a18058284b0 (127.0.97.61:45041)
I20260619 13:14:22.574945 7848 leader_election.cc:288] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3e798ed377b4da7ab9a1fbdc08cf6fe (127.0.97.62:45357), 13c24fa7041b4956a8601b23968dd506 (127.0.97.60:42585)
I20260619 13:14:22.577781 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:22.575066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:41078)
0619 13:14:22.575773 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:22.575953 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":399,"threads_started":1}
I20260619 13:14:22.577945 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:22.573993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:41068)
0619 13:14:22.577578 (+ 3585us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:22.577732 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3480,"threads_started":1}
W20260619 13:14:22.578799 7107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
W20260619 13:14:22.579514 7296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
I20260619 13:14:22.581307 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:22.580770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:41092)
0619 13:14:22.580961 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:22.581115 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 13:14:22.581568 7296 leader_election.cc:334] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a3e798ed377b4da7ab9a1fbdc08cf6fe (127.0.97.62:45357): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
I20260619 13:14:22.581995 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:22.581344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:41104)
0619 13:14:22.581592 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:22.581738 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260619 13:14:22.582185 7107 leader_election.cc:334] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a3e798ed377b4da7ab9a1fbdc08cf6fe (127.0.97.62:45357): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
I20260619 13:14:22.585011 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:22.576367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.0.97.60:42585)
0619 13:14:22.577463 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:22.577469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:22.578973 (+ 1504us) server_negotiation.cc:407] Connection header received
0619 13:14:22.579048 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:22.579053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:22.579126 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:22.579235 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:22.579764 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:22.580788 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:22.582887 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:22.584039 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:22.584166 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:22.584262 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:22.584635 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 13:14:22.584750 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962,"thread_start_us":831,"threads_started":1}
I20260619 13:14:22.585115 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:22.575831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:42585 (local address 127.0.0.1:38164)
0619 13:14:22.578168 (+ 2337us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:22.578188 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:22.578285 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:22.579276 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:22.579281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:22.579302 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:22.579555 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:22.579565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:22.580957 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:22.580962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:22.582751 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:22.582764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:22.584215 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:22.584219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:22.584429 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:22.584458 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 13:14:22.584610 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 13:14:22.584771 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2234,"mutex_wait_us":33,"thread_start_us":2617,"threads_started":1}
I20260619 13:14:22.585453 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:22.574172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39732 (local address 127.0.97.61:45041)
0619 13:14:22.578844 (+ 4672us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:22.578848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:22.578862 (+ 14us) server_negotiation.cc:407] Connection header received
0619 13:14:22.578914 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:22.578917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:22.578968 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:22.579054 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:22.579555 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:22.581170 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:22.583197 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:22.584405 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:22.584512 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:22.584611 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:22.585013 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 13:14:22.585144 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":4576,"thread_start_us":4437,"threads_started":1}
I20260619 13:14:22.585741 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:22.574227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45041 (local address 127.0.0.1:39732)
0619 13:14:22.578103 (+ 3876us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:22.578119 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:22.578252 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:22.579090 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:22.579095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:22.579133 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:22.579387 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:22.579399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:22.581336 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:22.581340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:22.583049 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:22.583060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:22.584650 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:22.584657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:22.584846 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:22.584884 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 13:14:22.584979 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 13:14:22.585424 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3796,"thread_start_us":3251,"threads_started":1}
I20260619 13:14:22.585800 7321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c7b3f0d5a26443f84775a18058284b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "13c24fa7041b4956a8601b23968dd506" is_pre_election: true
I20260619 13:14:22.586172 7321 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c7b3f0d5a26443f84775a18058284b0 in term 1.
I20260619 13:14:22.586385 7131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13c24fa7041b4956a8601b23968dd506" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5c7b3f0d5a26443f84775a18058284b0" is_pre_election: true
I20260619 13:14:22.586643 7107 leader_election.cc:302] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13c24fa7041b4956a8601b23968dd506, 5c7b3f0d5a26443f84775a18058284b0; no voters: a3e798ed377b4da7ab9a1fbdc08cf6fe
I20260619 13:14:22.586691 7131 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c24fa7041b4956a8601b23968dd506 in term 1.
I20260619 13:14:22.586969 7848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 13:14:22.587100 7848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a3e798ed377b4da7ab9a1fbdc08cf6fe)
I20260619 13:14:22.587198 7848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:14:22.587159 7297 leader_election.cc:302] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13c24fa7041b4956a8601b23968dd506, 5c7b3f0d5a26443f84775a18058284b0; no voters: a3e798ed377b4da7ab9a1fbdc08cf6fe
I20260619 13:14:22.587499 7847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 13:14:22.587620 7847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a3e798ed377b4da7ab9a1fbdc08cf6fe)
I20260619 13:14:22.587706 7847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:14:22.589331 7848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } }
I20260619 13:14:22.589674 7847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } }
I20260619 13:14:22.589970 7848 leader_election.cc:288] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [CANDIDATE]: Term 2 election: Requested vote from peers a3e798ed377b4da7ab9a1fbdc08cf6fe (127.0.97.62:45357), 13c24fa7041b4956a8601b23968dd506 (127.0.97.60:42585)
I20260619 13:14:22.590301 7847 leader_election.cc:288] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [CANDIDATE]: Term 2 election: Requested vote from peers a3e798ed377b4da7ab9a1fbdc08cf6fe (127.0.97.62:45357), 5c7b3f0d5a26443f84775a18058284b0 (127.0.97.61:45041)
I20260619 13:14:22.590420 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:22.590061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:41120)
0619 13:14:22.590200 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:22.590275 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 13:14:22.590586 7321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c7b3f0d5a26443f84775a18058284b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "13c24fa7041b4956a8601b23968dd506"
I20260619 13:14:22.590871 7321 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5c7b3f0d5a26443f84775a18058284b0 in current term 2: Already voted for candidate 13c24fa7041b4956a8601b23968dd506 in this term.
I20260619 13:14:22.591171 7131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13c24fa7041b4956a8601b23968dd506" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5c7b3f0d5a26443f84775a18058284b0"
I20260619 13:14:22.591450 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:22.591055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:41142)
0619 13:14:22.591229 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:22.591307 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 13:14:22.591475 7131 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 13c24fa7041b4956a8601b23968dd506 in current term 2: Already voted for candidate 5c7b3f0d5a26443f84775a18058284b0 in this term.
I20260619 13:14:22.591619 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:22.590678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:41130)
0619 13:14:22.590847 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:22.590959 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 13:14:22.591848 7107 leader_election.cc:334] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a3e798ed377b4da7ab9a1fbdc08cf6fe (127.0.97.62:45357): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
I20260619 13:14:22.591984 7107 leader_election.cc:302] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5c7b3f0d5a26443f84775a18058284b0; no voters: 13c24fa7041b4956a8601b23968dd506, a3e798ed377b4da7ab9a1fbdc08cf6fe
I20260619 13:14:22.592809 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:22.592445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:41162)
0619 13:14:22.592576 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:22.592673 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 13:14:22.592892 7848 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260619 13:14:22.593055 7296 leader_election.cc:334] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a3e798ed377b4da7ab9a1fbdc08cf6fe (127.0.97.62:45357): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
I20260619 13:14:22.593168 7296 leader_election.cc:302] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 13c24fa7041b4956a8601b23968dd506; no voters: 5c7b3f0d5a26443f84775a18058284b0, a3e798ed377b4da7ab9a1fbdc08cf6fe
I20260619 13:14:22.593405 7847 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 13:14:23.057811 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:23.057073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.1:38677)
0619 13:14:23.057520 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:23.057621 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":232,"threads_started":1}
I20260619 13:14:23.061489 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:23.060811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.2:45087)
0619 13:14:23.061265 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:23.061345 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":230,"threads_started":1}
I20260619 13:14:23.064142 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:23.063511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.3:50707)
0619 13:14:23.063895 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:23.063974 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":166,"threads_started":1}
I20260619 13:14:24.059306 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.058516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.1:55897)
0619 13:14:24.058978 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.059089 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":240,"threads_started":1}
I20260619 13:14:24.062719 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.062160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.2:44835)
0619 13:14:24.062497 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.062568 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":167,"threads_started":1}
I20260619 13:14:24.065428 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.064833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.3:54997)
0619 13:14:24.065227 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.065297 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":210,"threads_started":1}
I20260619 13:14:24.219553 7863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:14:24.219730 7863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } }
I20260619 13:14:24.220320 7863 leader_election.cc:288] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a3e798ed377b4da7ab9a1fbdc08cf6fe (127.0.97.62:45357), 5c7b3f0d5a26443f84775a18058284b0 (127.0.97.61:45041)
I20260619 13:14:24.220836 7131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13c24fa7041b4956a8601b23968dd506" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5c7b3f0d5a26443f84775a18058284b0" is_pre_election: true
I20260619 13:14:24.221067 7131 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c24fa7041b4956a8601b23968dd506 in term 2.
I20260619 13:14:24.221232 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.220510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:50674)
0619 13:14:24.220930 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.221012 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":201,"threads_started":1}
I20260619 13:14:24.221486 7297 leader_election.cc:302] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c24fa7041b4956a8601b23968dd506, 5c7b3f0d5a26443f84775a18058284b0; no voters:
I20260619 13:14:24.221719 7863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 13:14:24.221827 7863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:14:24.221961 7863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 2 FOLLOWER]: Advancing to term 3
I20260619 13:14:24.222572 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.222224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:50678)
0619 13:14:24.222367 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.222445 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 13:14:24.222788 7296 leader_election.cc:334] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a3e798ed377b4da7ab9a1fbdc08cf6fe (127.0.97.62:45357): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
I20260619 13:14:24.224112 7863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } }
I20260619 13:14:24.224699 7863 leader_election.cc:288] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [CANDIDATE]: Term 3 election: Requested vote from peers a3e798ed377b4da7ab9a1fbdc08cf6fe (127.0.97.62:45357), 5c7b3f0d5a26443f84775a18058284b0 (127.0.97.61:45041)
I20260619 13:14:24.225250 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.224762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:50684)
0619 13:14:24.224952 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.225033 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 13:14:24.225214 7131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13c24fa7041b4956a8601b23968dd506" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5c7b3f0d5a26443f84775a18058284b0"
I20260619 13:14:24.225440 7131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 2 FOLLOWER]: Advancing to term 3
I20260619 13:14:24.228411 7131 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c24fa7041b4956a8601b23968dd506 in term 3.
I20260619 13:14:24.228801 7297 leader_election.cc:302] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c24fa7041b4956a8601b23968dd506, 5c7b3f0d5a26443f84775a18058284b0; no voters:
I20260619 13:14:24.229065 7863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 13:14:24.229267 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.225825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:50692)
0619 13:14:24.229065 (+ 3240us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.229142 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2763}
W20260619 13:14:24.229398 7296 leader_election.cc:334] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a3e798ed377b4da7ab9a1fbdc08cf6fe (127.0.97.62:45357): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
I20260619 13:14:24.229674 7863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [term 3 LEADER]: Becoming Leader. State: Replica: 13c24fa7041b4956a8601b23968dd506, State: Running, Role: LEADER
I20260619 13:14:24.230068 7863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } }
I20260619 13:14:24.231777 7866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13c24fa7041b4956a8601b23968dd506. Latest consensus state: current_term: 3 leader_uuid: "13c24fa7041b4956a8601b23968dd506" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } } }
I20260619 13:14:24.232053 7866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:24.232697 7868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:14:24.236263 7868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:14:24.237018 7868 catalog_manager.cc:1259] Loaded cluster ID: 260ce7232d764c108a9ed51331674457
I20260619 13:14:24.237083 7868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:14:24.238186 7868 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:14:24.238934 7868 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: Loaded TSK: 0
I20260619 13:14:24.239702 7868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:14:24.283215 7310 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:14:24.289747 7310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:14:24.293589 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.293198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:50718)
0619 13:14:24.293383 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.293454 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 13:14:24.293618 7131 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [term 3 FOLLOWER]: Refusing update from remote peer 13c24fa7041b4956a8601b23968dd506: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 13:14:24.294240 7866 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 13:14:24.294574 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.294200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:50730)
0619 13:14:24.294364 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.294427 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 13:14:24.294826 7296 consensus_peers.cc:487] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 -> Peer a3e798ed377b4da7ab9a1fbdc08cf6fe (127.0.97.62:45357): Couldn't send request to peer a3e798ed377b4da7ab9a1fbdc08cf6fe. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 13:14:24.296542 7870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13c24fa7041b4956a8601b23968dd506. Latest consensus state: current_term: 3 leader_uuid: "13c24fa7041b4956a8601b23968dd506" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } } }
I20260619 13:14:24.296774 7870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:14:24.298147 7866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "13c24fa7041b4956a8601b23968dd506" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } } }
I20260619 13:14:24.298446 7866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:24.299127 7870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "13c24fa7041b4956a8601b23968dd506" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a3e798ed377b4da7ab9a1fbdc08cf6fe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45357 } } peers { permanent_uuid: "5c7b3f0d5a26443f84775a18058284b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45041 } attrs { promote: false } } peers { permanent_uuid: "13c24fa7041b4956a8601b23968dd506" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 42585 } attrs { promote: false } } }
I20260619 13:14:24.299461 7870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c7b3f0d5a26443f84775a18058284b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:14:24.328745 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.305585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43431 (local address 127.0.0.1:52970)
0619 13:14:24.305713 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.305731 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:24.305861 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:24.322658 (+ 16797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:24.322663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:24.322690 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:24.323095 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:24.323111 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.324552 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.324556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:24.326112 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:24.326123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.327988 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.327994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:24.328191 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:24.328224 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:14:24.328308 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 13:14:24.328437 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":44}
I20260619 13:14:24.328753 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.306981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.0.97.3:45933)
0619 13:14:24.322465 (+ 15484us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:24.322469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:24.322484 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:14:24.322540 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:24.322543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:24.322602 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:24.322691 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:24.323274 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.323986 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.326508 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.327701 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.327824 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:24.327929 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:24.328313 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 13:14:24.328450 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15386,"thread_start_us":15843,"threads_started":1}
I20260619 13:14:24.328814 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.305678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52970 (local address 127.0.97.1:43431)
0619 13:14:24.322241 (+ 16563us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:24.322248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:24.322268 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:14:24.322340 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:24.322347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:24.322429 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:24.323248 (+ 819us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:24.323442 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.324415 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.326285 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.327769 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.328186 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:24.328281 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:24.328327 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 13:14:24.328446 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16427,"thread_start_us":16836,"threads_started":1}
I20260619 13:14:24.329517 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.306626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45933 (local address 127.0.0.1:35542)
0619 13:14:24.318518 (+ 11892us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.318538 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:24.318661 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:24.322729 (+ 4068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:24.322734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:24.322771 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:24.323117 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:24.323127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.324116 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.324121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:24.326362 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:24.326378 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.327927 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.327937 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:24.328111 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:24.328171 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 13:14:24.328995 (+ 824us) client_negotiation.cc:240] Negotiation successful
0619 13:14:24.329136 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11751,"mutex_wait_us":2,"thread_start_us":12128,"threads_started":1}
I20260619 13:14:24.329828 7424 tablet_service.cc:1467] Processing CreateTablet for tablet aef649bccd194690a815d7c072eb8862 (DEFAULT_TABLE table=test-table [id=3ad5141e31f64aebbc774ac0de82f716]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:24.330121 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.306077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36241 (local address 127.0.0.1:36838)
0619 13:14:24.306542 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.306563 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:24.306659 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:24.324215 (+ 17556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:24.324220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:24.324249 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:24.324659 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:24.324670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.326149 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.326155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:24.327862 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:24.327878 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.329486 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.329492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:24.329654 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:24.329684 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 13:14:24.329741 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 13:14:24.329879 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":264,"threads_started":1}
I20260619 13:14:24.330276 7424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef649bccd194690a815d7c072eb8862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:24.330397 7711 tablet_service.cc:1467] Processing CreateTablet for tablet aef649bccd194690a815d7c072eb8862 (DEFAULT_TABLE table=test-table [id=3ad5141e31f64aebbc774ac0de82f716]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:24.330790 7711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef649bccd194690a815d7c072eb8862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:24.330863 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.306120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.0.97.2:36241)
0619 13:14:24.323939 (+ 17819us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:24.323945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:24.323963 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:14:24.324022 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:24.324027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:24.324092 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:24.324187 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:24.324914 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.326014 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.328073 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.329292 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.330337 (+ 1045us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:24.330425 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:24.330472 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 13:14:24.330591 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17693,"thread_start_us":18170,"threads_started":1}
I20260619 13:14:24.331534 7570 tablet_service.cc:1467] Processing CreateTablet for tablet aef649bccd194690a815d7c072eb8862 (DEFAULT_TABLE table=test-table [id=3ad5141e31f64aebbc774ac0de82f716]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:24.331892 7570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef649bccd194690a815d7c072eb8862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:24.335614 7881 tablet_bootstrap.cc:492] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164: Bootstrap starting.
I20260619 13:14:24.337054 7882 tablet_bootstrap.cc:492] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9: Bootstrap starting.
I20260619 13:14:24.337224 7881 tablet_bootstrap.cc:654] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:24.337883 7883 tablet_bootstrap.cc:492] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8: Bootstrap starting.
I20260619 13:14:24.338780 7882 tablet_bootstrap.cc:654] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:24.339223 7883 tablet_bootstrap.cc:654] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:24.340700 7882 tablet_bootstrap.cc:492] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9: No bootstrap required, opened a new log
I20260619 13:14:24.340842 7882 ts_tablet_manager.cc:1397] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 13:14:24.340903 7883 tablet_bootstrap.cc:492] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8: No bootstrap required, opened a new log
I20260619 13:14:24.340917 7881 tablet_bootstrap.cc:492] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164: No bootstrap required, opened a new log
I20260619 13:14:24.341048 7881 ts_tablet_manager.cc:1397] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 13:14:24.341068 7883 ts_tablet_manager.cc:1397] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20260619 13:14:24.341609 7883 raft_consensus.cc:348] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } } peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } }
I20260619 13:14:24.341627 7882 raft_consensus.cc:348] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } } peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } }
I20260619 13:14:24.341797 7883 raft_consensus.cc:374] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:24.341756 7881 raft_consensus.cc:348] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } } peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } }
I20260619 13:14:24.341902 7883 raft_consensus.cc:729] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436161bbf1444a99a6611cda7569a9a8, State: Initialized, Role: FOLLOWER
I20260619 13:14:24.341934 7882 raft_consensus.cc:374] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:24.341981 7881 raft_consensus.cc:374] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:24.342073 7881 raft_consensus.cc:729] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d085b70f4c4d4185be1b6a426e622164, State: Initialized, Role: FOLLOWER
I20260619 13:14:24.342094 7882 raft_consensus.cc:729] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410babf59db6434c9d72b772ac83f5f9, State: Initialized, Role: FOLLOWER
I20260619 13:14:24.342097 7883 consensus_queue.cc:260] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } } peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } }
I20260619 13:14:24.342314 7882 consensus_queue.cc:260] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } } peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } }
I20260619 13:14:24.342314 7881 consensus_queue.cc:260] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } } peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } }
I20260619 13:14:24.343833 7883 ts_tablet_manager.cc:1428] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 13:14:24.345175 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.344428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.2:42513)
0619 13:14:24.344958 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.345037 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432}
I20260619 13:14:24.346750 7635 heartbeater.cc:503] Master 127.0.97.60:42585 was elected leader, sending a full tablet report...
I20260619 13:14:24.348026 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.347378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.2:40595)
0619 13:14:24.347802 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.347883 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
I20260619 13:14:24.350345 7882 ts_tablet_manager.cc:1428] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9: Time spent starting tablet: real 0.009s user 0.003s sys 0.006s
I20260619 13:14:24.350625 7881 ts_tablet_manager.cc:1428] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164: Time spent starting tablet: real 0.009s user 0.003s sys 0.007s
I20260619 13:14:24.351410 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.350903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.1:45453)
0619 13:14:24.351143 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.351238 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260619 13:14:24.351735 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.351377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.3:48049)
0619 13:14:24.351514 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.351601 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 13:14:24.353103 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.352701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.1:58959)
0619 13:14:24.352898 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.352971 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260619 13:14:24.353181 7776 heartbeater.cc:503] Master 127.0.97.60:42585 was elected leader, sending a full tablet report...
I20260619 13:14:24.354111 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.352939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.3:49331)
0619 13:14:24.353889 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.353965 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":822}
I20260619 13:14:24.354230 7489 heartbeater.cc:503] Master 127.0.97.60:42585 was elected leader, sending a full tablet report...
I20260619 13:14:24.569219 7887 raft_consensus.cc:482] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:14:24.569445 7887 raft_consensus.cc:504] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } } peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } }
I20260619 13:14:24.570190 7887 leader_election.cc:288] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d085b70f4c4d4185be1b6a426e622164 (127.0.97.1:43431), 410babf59db6434c9d72b772ac83f5f9 (127.0.97.3:45933)
I20260619 13:14:24.570698 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef649bccd194690a815d7c072eb8862" candidate_uuid: "436161bbf1444a99a6611cda7569a9a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d085b70f4c4d4185be1b6a426e622164" is_pre_election: true
I20260619 13:14:24.570732 7731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef649bccd194690a815d7c072eb8862" candidate_uuid: "436161bbf1444a99a6611cda7569a9a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410babf59db6434c9d72b772ac83f5f9" is_pre_election: true
I20260619 13:14:24.570919 7444 raft_consensus.cc:2454] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 436161bbf1444a99a6611cda7569a9a8 in term 0.
I20260619 13:14:24.570968 7731 raft_consensus.cc:2454] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 436161bbf1444a99a6611cda7569a9a8 in term 0.
I20260619 13:14:24.571357 7526 leader_election.cc:302] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 436161bbf1444a99a6611cda7569a9a8, d085b70f4c4d4185be1b6a426e622164; no voters:
I20260619 13:14:24.571632 7887 raft_consensus.cc:2781] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:14:24.571738 7887 raft_consensus.cc:482] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:14:24.571825 7887 raft_consensus.cc:3037] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:24.573712 7887 raft_consensus.cc:504] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } } peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } }
I20260619 13:14:24.574247 7887 leader_election.cc:288] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [CANDIDATE]: Term 1 election: Requested vote from peers d085b70f4c4d4185be1b6a426e622164 (127.0.97.1:43431), 410babf59db6434c9d72b772ac83f5f9 (127.0.97.3:45933)
I20260619 13:14:24.574625 7444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef649bccd194690a815d7c072eb8862" candidate_uuid: "436161bbf1444a99a6611cda7569a9a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d085b70f4c4d4185be1b6a426e622164"
I20260619 13:14:24.574723 7731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef649bccd194690a815d7c072eb8862" candidate_uuid: "436161bbf1444a99a6611cda7569a9a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "410babf59db6434c9d72b772ac83f5f9"
I20260619 13:14:24.574810 7444 raft_consensus.cc:3037] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:24.574916 7731 raft_consensus.cc:3037] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:24.577538 7444 raft_consensus.cc:2454] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436161bbf1444a99a6611cda7569a9a8 in term 1.
I20260619 13:14:24.577539 7731 raft_consensus.cc:2454] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436161bbf1444a99a6611cda7569a9a8 in term 1.
I20260619 13:14:24.577939 7525 leader_election.cc:302] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 410babf59db6434c9d72b772ac83f5f9, 436161bbf1444a99a6611cda7569a9a8, d085b70f4c4d4185be1b6a426e622164; no voters:
I20260619 13:14:24.578209 7887 raft_consensus.cc:2781] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:24.578572 7887 raft_consensus.cc:686] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [term 1 LEADER]: Becoming Leader. State: Replica: 436161bbf1444a99a6611cda7569a9a8, State: Running, Role: LEADER
I20260619 13:14:24.578831 7887 consensus_queue.cc:237] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } } peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } }
I20260619 13:14:24.579449 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.579088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.2:39177)
0619 13:14:24.579214 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.579314 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 13:14:24.581573 7309 catalog_manager.cc:5495] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 436161bbf1444a99a6611cda7569a9a8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "436161bbf1444a99a6611cda7569a9a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "436161bbf1444a99a6611cda7569a9a8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 } health_report { overall_health: UNKNOWN } } }
I20260619 13:14:24.605484 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.601286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.0.97.2:36241)
0619 13:14:24.601517 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:24.601524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:24.601542 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:14:24.601605 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:24.601610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:24.601668 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:24.601739 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:14:24.602096 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.602792 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.603991 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.604386 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.604488 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:24.604846 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:14:24.604952 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:14:24.605032 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:24.605164 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 13:14:24.605236 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 13:14:24.608333 7731 raft_consensus.cc:1261] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9 [term 1 FOLLOWER]: Refusing update from remote peer 436161bbf1444a99a6611cda7569a9a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:24.608333 7444 raft_consensus.cc:1261] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164 [term 1 FOLLOWER]: Refusing update from remote peer 436161bbf1444a99a6611cda7569a9a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:24.608894 7890 consensus_queue.cc:1035] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "410babf59db6434c9d72b772ac83f5f9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:24.609150 7887 consensus_queue.cc:1035] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d085b70f4c4d4185be1b6a426e622164" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:24.611346 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.610823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.1:56395)
0619 13:14:24.611063 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.611156 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260619 13:14:24.612002 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.611487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.3:55443)
0619 13:14:24.611705 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.611800 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260619 13:14:24.617754 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.617138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.1:50833)
0619 13:14:24.617468 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.617555 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260619 13:14:24.629658 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.629118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.2:34139)
0619 13:14:24.629411 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.629496 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260619 13:14:24.629738 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.627480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.3:42687)
0619 13:14:24.627878 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.627964 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283}
I20260619 13:14:24.630812 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.630377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.97.2:46845)
0619 13:14:24.630595 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.630672 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260619 13:14:24.634032 7309 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:24.634280 7309 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:24.638793 7309 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13c24fa7041b4956a8601b23968dd506: Sending DeleteTablet for 3 replicas of tablet aef649bccd194690a815d7c072eb8862
I20260619 13:14:24.639712 7570 tablet_service.cc:1514] Processing DeleteTablet for tablet aef649bccd194690a815d7c072eb8862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:24 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:36838
I20260619 13:14:24.639787 7711 tablet_service.cc:1514] Processing DeleteTablet for tablet aef649bccd194690a815d7c072eb8862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:24 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:35542
I20260619 13:14:24.639806 7424 tablet_service.cc:1514] Processing DeleteTablet for tablet aef649bccd194690a815d7c072eb8862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:24 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:52970
I20260619 13:14:24.640415 7902 tablet_replica.cc:330] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9: stopping tablet replica
I20260619 13:14:24.640551 7901 tablet_replica.cc:330] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8: stopping tablet replica
I20260619 13:14:24.640592 7902 raft_consensus.cc:2229] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:24.640708 7901 raft_consensus.cc:2229] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:14:24.640856 7902 raft_consensus.cc:2258] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:24.641068 7901 raft_consensus.cc:2258] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:24.641376 7903 tablet_replica.cc:330] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164: stopping tablet replica
I20260619 13:14:24.641542 7903 raft_consensus.cc:2229] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:24.641781 7903 raft_consensus.cc:2258] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:24.641775 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 7319
I20260619 13:14:24.642084 7901 ts_tablet_manager.cc:1905] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:24.642650 7902 ts_tablet_manager.cc:1905] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:24.644681 7903 ts_tablet_manager.cc:1905] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:24.648183 7901 ts_tablet_manager.cc:1918] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:24.648319 7901 log.cc:1196] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/aef649bccd194690a815d7c072eb8862
I20260619 13:14:24.648543 7902 ts_tablet_manager.cc:1918] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:24.648663 7902 log.cc:1196] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/aef649bccd194690a815d7c072eb8862
I20260619 13:14:24.648705 7901 ts_tablet_manager.cc:1939] T aef649bccd194690a815d7c072eb8862 P 436161bbf1444a99a6611cda7569a9a8: Deleting consensus metadata
I20260619 13:14:24.649042 7902 ts_tablet_manager.cc:1939] T aef649bccd194690a815d7c072eb8862 P 410babf59db6434c9d72b772ac83f5f9: Deleting consensus metadata
I20260619 13:14:24.649772 7296 catalog_manager.cc:4841] TS 436161bbf1444a99a6611cda7569a9a8 (127.0.97.2:36241): tablet aef649bccd194690a815d7c072eb8862 (table test-table [id=3ad5141e31f64aebbc774ac0de82f716]) successfully deleted
I20260619 13:14:24.650103 7296 catalog_manager.cc:4841] TS 410babf59db6434c9d72b772ac83f5f9 (127.0.97.3:45933): tablet aef649bccd194690a815d7c072eb8862 (table test-table [id=3ad5141e31f64aebbc774ac0de82f716]) successfully deleted
I20260619 13:14:24.651424 7903 ts_tablet_manager.cc:1918] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:24.651551 7903 log.cc:1196] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/aef649bccd194690a815d7c072eb8862
I20260619 13:14:24.651968 7903 ts_tablet_manager.cc:1939] T aef649bccd194690a815d7c072eb8862 P d085b70f4c4d4185be1b6a426e622164: Deleting consensus metadata
I20260619 13:14:24.652936 7297 catalog_manager.cc:4841] TS d085b70f4c4d4185be1b6a426e622164 (127.0.97.1:43431): tablet aef649bccd194690a815d7c072eb8862 (table test-table [id=3ad5141e31f64aebbc774ac0de82f716]) successfully deleted
I20260619 13:14:24.667511 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.644121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.0.97.1:43431)
0619 13:14:24.644261 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:24.644268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:24.653245 (+ 8977us) server_negotiation.cc:407] Connection header received
0619 13:14:24.653400 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:24.653404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:24.653483 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:24.653573 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:24.654754 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.655401 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.656542 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.656938 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.657026 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:24.662637 (+ 5611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:24.662651 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:24.662666 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:24.662702 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:24.665332 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:24.665959 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:24.665966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:24.665968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:24.666085 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:24.666380 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:24.666383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:24.666385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:24.666710 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:24.666913 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:24.667118 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 13:14:24.667277 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 13:14:24.668113 7484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:24.678551 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.678112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:50732)
0619 13:14:24.678283 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.678384 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 13:14:24.679507 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.679130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:50734)
0619 13:14:24.679268 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:24.679357 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 13:14:24.758010 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 7319
I20260619 13:14:24.772220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 7501
I20260619 13:14:24.784412 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.772881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.0.97.2:36241)
0619 13:14:24.773039 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:24.773044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:24.773115 (+ 71us) server_negotiation.cc:407] Connection header received
0619 13:14:24.773307 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:24.773311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:24.773382 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:24.773464 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:24.774731 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.775457 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.776601 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.776991 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.777109 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:24.779386 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:24.779400 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:24.779416 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:24.779447 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:24.782017 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:24.782620 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:24.782627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:24.782629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:24.782740 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:24.783034 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:24.783038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:24.783042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:24.783393 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:24.783615 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:24.783935 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 13:14:24.784134 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 13:14:24.785144 7630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:24.902485 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 7501
I20260619 13:14:24.919489 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 7648
I20260619 13:14:24.931963 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:24.920170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.0.97.3:45933)
0619 13:14:24.920582 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:24.920587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:24.920603 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:14:24.920656 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:24.920660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:24.920730 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:24.920819 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:24.922104 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.922850 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.924092 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:24.924543 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:24.924640 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:24.927177 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:24.927192 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:24.927209 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:24.927267 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:24.929827 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:24.930418 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:24.930425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:24.930427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:24.930569 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:24.930871 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:24.930874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:24.930876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:24.931179 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:24.931396 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:24.931572 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 13:14:24.931721 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":220,"threads_started":1}
I20260619 13:14:24.932536 7771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:25.026813 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 7648
I20260619 13:14:25.040369 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 6985
I20260619 13:14:25.041394 7151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:25.154439 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 6985
I20260619 13:14:25.155591 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:25.155211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:50738)
0619 13:14:25.155366 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:25.155450 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 13:14:25.156584 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:25.156280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45357 (local address 127.0.0.1:50740)
0619 13:14:25.156415 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:25.156481 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 13:14:25.170483 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 7085
I20260619 13:14:25.171416 7341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:25.285893 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 7085
2026-06-19T13:14:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10569 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:14:25 dist-test-slave-mkt3 krb5kdc[7913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:14:25 dist-test-slave-mkt3 krb5kdc[7913](info): set up 2 sockets
Jun 19 13:14:25 dist-test-slave-mkt3 krb5kdc[7913](info): commencing operation
krb5kdc: starting...
W20260619 13:14:26.286298 388 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.009s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:14:26 dist-test-slave-mkt3 krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:14:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:14:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260619 13:14:26.470628 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40309
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:26.771811 7926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:26.772076 7926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:26.772167 7926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:26.779157 7926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:26.779264 7926 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 13:14:26.779313 7926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:26.779371 7926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:26.779418 7926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:26.779475 7926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:14:26.788967 7926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40309
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:26.790134 7926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:26.791352 7926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:26.802198 7931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:26.802264 7932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:26.802295 7934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:26.803113 7926 server_base.cc:1034] running on GCE node
I20260619 13:14:26.803791 7926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:26.805063 7926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:26.806273 7926 hybrid_clock.cc:648] HybridClock initialized: now 1781874866806198 us; error 48 us; skew 500 ppm
I20260619 13:14:26.806619 7926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:26 dist-test-slave-mkt3 krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:26.809471 7926 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260619 13:14:26.810853 7926 webserver.cc:460] Webserver started at http://127.0.97.62:40049/ using document root <none> and password file <none>
I20260619 13:14:26.811515 7926 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:26.811615 7926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:26.811904 7926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:26.814733 7926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "924af2bae47e454798718f65b6b38e7d"
format_stamp: "Formatted at 2026-06-19 13:14:26 on dist-test-slave-mkt3"
server_key: "7344db6f789eab33a4f8389385bb079a"
server_key_iv: "574171aee57b6d3decb1bcfc132005af"
server_key_version: "encryptionkey@0"
I20260619 13:14:26.815461 7926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "924af2bae47e454798718f65b6b38e7d"
format_stamp: "Formatted at 2026-06-19 13:14:26 on dist-test-slave-mkt3"
server_key: "7344db6f789eab33a4f8389385bb079a"
server_key_iv: "574171aee57b6d3decb1bcfc132005af"
server_key_version: "encryptionkey@0"
I20260619 13:14:26.820266 7926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:14:26.823457 7941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:26.824704 7926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 13:14:26.824841 7926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "924af2bae47e454798718f65b6b38e7d"
format_stamp: "Formatted at 2026-06-19 13:14:26 on dist-test-slave-mkt3"
server_key: "7344db6f789eab33a4f8389385bb079a"
server_key_iv: "574171aee57b6d3decb1bcfc132005af"
server_key_version: "encryptionkey@0"
I20260619 13:14:26.825024 7926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:26.847465 7926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:26.851011 7926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:26.851233 7926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:26.870194 7926 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40309
I20260619 13:14:26.870224 8002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40309 every 8 connection(s)
I20260619 13:14:26.871407 7926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:14:26.874925 8003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:26.877816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 7926
I20260619 13:14:26.878031 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:14:26.878620 388 external_mini_cluster.cc:1428] Setting key 596ef14552b481198ed212b9af912db0
I20260619 13:14:26.886022 8003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d: Bootstrap starting.
I20260619 13:14:26.888815 8003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:26.889776 8003 log.cc:824] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:14:26 dist-test-slave-mkt3 krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:14:26.892601 8003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d: No bootstrap required, opened a new log
I20260619 13:14:26.899462 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:26.880515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39416 (local address 127.0.97.62:40309)
0619 13:14:26.881233 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:26.881244 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:26.881283 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:14:26.882904 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:26.882929 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:26.883324 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:26.883675 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:26.885020 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:26.886362 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:26.887570 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:26.888211 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:26.888366 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:26.892521 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:26.892529 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:26.892543 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:26.892593 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:26.895605 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:26.896384 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:26.896391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:26.896397 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:26.896553 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:26.896949 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:26.896953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:26.896955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:26.897456 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:26.897672 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:26.898114 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 13:14:26.898446 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":310,"threads_started":1}
I20260619 13:14:26.900925 8003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924af2bae47e454798718f65b6b38e7d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40309 } }
I20260619 13:14:26.901355 8003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:26.901461 8003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924af2bae47e454798718f65b6b38e7d, State: Initialized, Role: FOLLOWER
I20260619 13:14:26.902091 8003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924af2bae47e454798718f65b6b38e7d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40309 } }
I20260619 13:14:26.902310 8003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:14:26.902428 8003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:14:26.902576 8003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:26.904726 8003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924af2bae47e454798718f65b6b38e7d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40309 } }
I20260619 13:14:26.905174 8003 leader_election.cc:302] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 924af2bae47e454798718f65b6b38e7d; no voters:
I20260619 13:14:26.905647 8003 leader_election.cc:288] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:14:26.905752 8008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:26.906468 8008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [term 1 LEADER]: Becoming Leader. State: Replica: 924af2bae47e454798718f65b6b38e7d, State: Running, Role: LEADER
I20260619 13:14:26.906955 8008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924af2bae47e454798718f65b6b38e7d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40309 } }
I20260619 13:14:26.907449 8003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:26.909458 8009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "924af2bae47e454798718f65b6b38e7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924af2bae47e454798718f65b6b38e7d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40309 } } }
I20260619 13:14:26.909332 8010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 924af2bae47e454798718f65b6b38e7d. Latest consensus state: current_term: 1 leader_uuid: "924af2bae47e454798718f65b6b38e7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924af2bae47e454798718f65b6b38e7d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40309 } } }
I20260619 13:14:26.909761 8009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:26.909758 8010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:26.911262 8015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:14:26.915923 8015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:14:26.924921 8015 catalog_manager.cc:1347] Generated new cluster ID: 6f839c4cece34d3f839ac9d5d7f7819c
I20260619 13:14:26.925025 8015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:14:26.935842 8015 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:14:26.937274 8015 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:14:26.946568 8015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d: Generated new TSK 0
I20260619 13:14:26.947515 8015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 13:14:26.998967 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40309
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:27.306771 8029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:27.307035 8029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:27.307168 8029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:27.314107 8029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:27.314201 8029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:27.314314 8029 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:14:27.314401 8029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:14:27.324210 8029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:27.325371 8029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:27.326718 8029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:27.338415 8034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:27.338469 8035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:27.338500 8037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:27.339202 8029 server_base.cc:1034] running on GCE node
I20260619 13:14:27.339716 8029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:27.340518 8029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:27.341714 8029 hybrid_clock.cc:648] HybridClock initialized: now 1781874867341656 us; error 36 us; skew 500 ppm
I20260619 13:14:27.341992 8029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:27 dist-test-slave-mkt3 krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:27.344833 8029 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 13:14:27.346199 8029 webserver.cc:460] Webserver started at http://127.0.97.1:34911/ using document root <none> and password file <none>
I20260619 13:14:27.346828 8029 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:27.346920 8029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:27.347198 8029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:27.349884 8029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "cf8178fb2a7140ca9a532e3ebb39dd04"
format_stamp: "Formatted at 2026-06-19 13:14:27 on dist-test-slave-mkt3"
server_key: "f222716bc6c9c95ffb6beeddd6b4c03b"
server_key_iv: "a96fe3df8784877f3984067aa90ded3c"
server_key_version: "encryptionkey@0"
I20260619 13:14:27.350618 8029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cf8178fb2a7140ca9a532e3ebb39dd04"
format_stamp: "Formatted at 2026-06-19 13:14:27 on dist-test-slave-mkt3"
server_key: "f222716bc6c9c95ffb6beeddd6b4c03b"
server_key_iv: "a96fe3df8784877f3984067aa90ded3c"
server_key_version: "encryptionkey@0"
I20260619 13:14:27.355268 8029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 13:14:27.358525 8044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:27.359752 8029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:14:27.359894 8029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "cf8178fb2a7140ca9a532e3ebb39dd04"
format_stamp: "Formatted at 2026-06-19 13:14:27 on dist-test-slave-mkt3"
server_key: "f222716bc6c9c95ffb6beeddd6b4c03b"
server_key_iv: "a96fe3df8784877f3984067aa90ded3c"
server_key_version: "encryptionkey@0"
I20260619 13:14:27.360077 8029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:27.378471 8029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:27.381946 8029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:27.382184 8029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:27.384605 8029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:27.384706 8029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:27.384872 8029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:27.384939 8029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:14:27 dist-test-slave-mkt3 krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:14:27.405754 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:27.385002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43249 (local address 127.0.97.62:40309)
0619 13:14:27.385143 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:27.385148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:27.386454 (+ 1306us) server_negotiation.cc:407] Connection header received
0619 13:14:27.388419 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:27.388425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:27.388509 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:27.388624 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:27.390549 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.391640 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.393904 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.394532 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.394659 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:27.398910 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:27.398917 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:27.398922 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:27.398961 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:27.402281 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:27.403132 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:27.403138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:27.403143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:27.403300 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:27.403855 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:27.403861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:27.403864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:27.404205 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:27.404349 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:27.405225 (+ 876us) server_negotiation.cc:299] Negotiation successful
0619 13:14:27.405429 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 13:14:27.406785 8053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:27.385582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40309 (local address 127.0.97.1:43249)
0619 13:14:27.386307 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:27.386341 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:27.388089 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:27.388864 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:27.388874 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:27.389570 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:27.390361 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:27.390388 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.391842 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.391850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:27.393729 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:27.393755 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.394789 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.394797 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:27.394974 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:27.395721 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:27.395768 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:27.398683 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:27.402408 (+ 3725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:27.402418 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:27.402438 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:27.402987 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:27.403422 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:27.403427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:27.403431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:27.403726 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:27.404406 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:27.404412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:27.404876 (+ 464us) client_negotiation.cc:769] Sending connection context
0619 13:14:27.405113 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 13:14:27.405626 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":324,"threads_started":1}
I20260619 13:14:27.429548 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:27.416033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40309 (local address 127.0.97.1:46437)
0619 13:14:27.416668 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:27.416685 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:27.416794 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:27.417076 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:27.417079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:27.417630 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:27.418088 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:27.418105 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.419341 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.419345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:27.420879 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:27.420894 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.421766 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.421771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:27.421928 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:27.422508 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:27.422532 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:27.423496 (+ 964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:27.426890 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:27.426896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:27.426900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:27.427437 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:27.427801 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:27.427806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:27.427810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:27.427944 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:27.428614 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:27.428617 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:27.428778 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 13:14:27.428886 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 13:14:27.429169 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":340,"threads_started":1}
I20260619 13:14:27.429636 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:27.416034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46437 (local address 127.0.97.62:40309)
0619 13:14:27.416224 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:27.416231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:27.416751 (+ 520us) server_negotiation.cc:407] Connection header received
0619 13:14:27.416881 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:27.416886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:27.416962 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:27.417070 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:27.418253 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.419191 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.421027 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.421608 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.421735 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:27.423726 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:27.423732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:27.423737 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:27.423775 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:27.426768 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:27.427543 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:27.427548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:27.427552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:27.427683 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:27.428072 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:27.428077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:27.428080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:27.428475 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:27.428663 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:27.429020 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 13:14:27.429258 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:14:27.431985 8029 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44349
I20260619 13:14:27.432001 8163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44349 every 8 connection(s)
I20260619 13:14:27.433324 8029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:14:27.435159 8164 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40309
I20260619 13:14:27.435348 8164 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:27.435909 8164 heartbeater.cc:511] Master 127.0.97.62:40309 requested a full tablet report, sending...
I20260619 13:14:27.437671 7958 ts_manager.cc:194] Registered new tserver with Master: cf8178fb2a7140ca9a532e3ebb39dd04 (127.0.97.1:44349)
I20260619 13:14:27.438114 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 8029
I20260619 13:14:27.438287 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:14:27.438844 388 external_mini_cluster.cc:1428] Setting key d8085b41ece3e375d141c4f7fc9eea11
I20260619 13:14:27.439244 7958 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43249
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260619 13:14:27.478718 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40309
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:27.777982 8168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:27.778232 8168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:27.778323 8168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:27.785194 8168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:27.785283 8168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:27.785368 8168 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:14:27.785444 8168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:14:27.794443 8168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:27.795495 8168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:27.796706 8168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:27.808753 8173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:27.808806 8176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:27.808826 8174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:27.809564 8168 server_base.cc:1034] running on GCE node
I20260619 13:14:27.810201 8168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:27.811172 8168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:27.812378 8168 hybrid_clock.cc:648] HybridClock initialized: now 1781874867812307 us; error 50 us; skew 500 ppm
I20260619 13:14:27.812654 8168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:27 dist-test-slave-mkt3 krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:27.815495 8168 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 13:14:27.816869 8168 webserver.cc:460] Webserver started at http://127.0.97.2:34515/ using document root <none> and password file <none>
I20260619 13:14:27.817521 8168 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:27.817615 8168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:27.817943 8168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:27.820691 8168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "f33f57b01113407a84cc1b6bd8a78883"
format_stamp: "Formatted at 2026-06-19 13:14:27 on dist-test-slave-mkt3"
server_key: "480e42aa2ff6d1c747f8c6a65bd716ed"
server_key_iv: "e941a7fa497a1f08f12bb98c06ee2405"
server_key_version: "encryptionkey@0"
I20260619 13:14:27.821480 8168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f33f57b01113407a84cc1b6bd8a78883"
format_stamp: "Formatted at 2026-06-19 13:14:27 on dist-test-slave-mkt3"
server_key: "480e42aa2ff6d1c747f8c6a65bd716ed"
server_key_iv: "e941a7fa497a1f08f12bb98c06ee2405"
server_key_version: "encryptionkey@0"
I20260619 13:14:27.826285 8168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260619 13:14:27.829493 8183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:27.830772 8168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:14:27.830909 8168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "f33f57b01113407a84cc1b6bd8a78883"
format_stamp: "Formatted at 2026-06-19 13:14:27 on dist-test-slave-mkt3"
server_key: "480e42aa2ff6d1c747f8c6a65bd716ed"
server_key_iv: "e941a7fa497a1f08f12bb98c06ee2405"
server_key_version: "encryptionkey@0"
I20260619 13:14:27.831089 8168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:27.852762 8168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:27.857357 8168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:27.857620 8168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:27.864462 8168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:27.864560 8168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:27.864692 8168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:27.864758 8168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:14:27 dist-test-slave-mkt3 krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874867, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:14:27.888058 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:27.865563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60851 (local address 127.0.97.62:40309)
0619 13:14:27.865729 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:27.865735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:27.867314 (+ 1579us) server_negotiation.cc:407] Connection header received
0619 13:14:27.870138 (+ 2824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:27.870144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:27.870227 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:27.870298 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:27.872407 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.873530 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.875963 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.876635 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.876725 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:27.881134 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:27.881141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:27.881146 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:27.881184 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:27.884444 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:27.885327 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:27.885333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:27.885337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:27.885473 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:27.886079 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:27.886084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:27.886087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:27.886438 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:27.886572 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:27.887465 (+ 893us) server_negotiation.cc:299] Negotiation successful
0619 13:14:27.887725 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 13:14:27.890673 8192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:27.866213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40309 (local address 127.0.97.2:60851)
0619 13:14:27.867138 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:27.867193 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:27.869799 (+ 2606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:27.870781 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:27.870793 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:27.871468 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:27.872226 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:27.872248 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.873736 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.873742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:27.875715 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:27.875730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.877026 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.877032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:27.877277 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:27.878024 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:27.878060 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:27.880868 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:27.884629 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:27.884637 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:27.884652 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:27.885194 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:27.885608 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:27.885613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:27.885617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:27.885868 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:27.886666 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:27.886674 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:27.887086 (+ 412us) client_negotiation.cc:769] Sending connection context
0619 13:14:27.888734 (+ 1648us) client_negotiation.cc:240] Negotiation successful
0619 13:14:27.889254 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":352,"threads_started":1}
I20260619 13:14:27.922724 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:27.906462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40309 (local address 127.0.97.2:46633)
0619 13:14:27.907717 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:27.907736 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:27.907876 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:27.908368 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:27.908373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:27.908962 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:27.909399 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:27.909418 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.910835 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.910840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:27.912653 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:27.912671 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.913820 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.913827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:27.914031 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:27.914851 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:27.914874 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:27.915852 (+ 978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:27.919713 (+ 3861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:27.919718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:27.919722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:27.920274 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:27.920689 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:27.920697 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:27.920701 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:27.920853 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:27.921584 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:27.921589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:27.921794 (+ 205us) client_negotiation.cc:769] Sending connection context
0619 13:14:27.921972 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 13:14:27.922270 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"thread_start_us":977,"threads_started":1}
I20260619 13:14:27.923244 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:27.906937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46633 (local address 127.0.97.62:40309)
0619 13:14:27.907098 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:27.907104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:27.907821 (+ 717us) server_negotiation.cc:407] Connection header received
0619 13:14:27.907997 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:27.908002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:27.908081 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:27.908168 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:27.909631 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.910641 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.912852 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:27.913438 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:27.913553 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:27.916073 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:27.916081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:27.916086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:27.916130 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:27.919568 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:27.920410 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:27.920416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:27.920420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:27.920559 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:27.920968 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:27.920972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:27.920975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:27.921385 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:27.921522 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:27.922724 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0619 13:14:27.922938 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:14:27.948339 8168 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36587
I20260619 13:14:27.948357 8302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36587 every 8 connection(s)
I20260619 13:14:27.949522 8168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:14:27.951467 8303 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40309
I20260619 13:14:27.951676 8303 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:27.952265 8303 heartbeater.cc:511] Master 127.0.97.62:40309 requested a full tablet report, sending...
I20260619 13:14:27.953712 7957 ts_manager.cc:194] Registered new tserver with Master: f33f57b01113407a84cc1b6bd8a78883 (127.0.97.2:36587)
I20260619 13:14:27.954846 7957 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60851
I20260619 13:14:27.957952 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 8168
I20260619 13:14:27.958151 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:14:27.958666 388 external_mini_cluster.cc:1428] Setting key 6224688005dcfbed6dd2ec8c71fd3cc7
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 13:14:27.997603 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40309
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:28.299523 8307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:28.299817 8307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:28.299942 8307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:28.306973 8307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:28.307111 8307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:28.307235 8307 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 13:14:28.307363 8307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:14:28.317245 8307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:40309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:28.318532 8307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:28.319794 8307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:28.332166 8313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:28.332134 8315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:28.332219 8312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:28.338361 8307 server_base.cc:1034] running on GCE node
I20260619 13:14:28.339004 8307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:28.339939 8307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:28.341166 8307 hybrid_clock.cc:648] HybridClock initialized: now 1781874868341096 us; error 46 us; skew 500 ppm
I20260619 13:14:28.341483 8307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:28 dist-test-slave-mkt3 krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874868, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:28.344628 8307 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 13:14:28.346119 8307 webserver.cc:460] Webserver started at http://127.0.97.3:40805/ using document root <none> and password file <none>
I20260619 13:14:28.346807 8307 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:28.346894 8307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:28.347203 8307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:28.350152 8307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "745e8c590de7481098df279ed860336f"
format_stamp: "Formatted at 2026-06-19 13:14:28 on dist-test-slave-mkt3"
server_key: "de03228dc28b325da8b71d960d942425"
server_key_iv: "206fff252983f3cd158cadf19fda2ce0"
server_key_version: "encryptionkey@0"
I20260619 13:14:28.350915 8307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "745e8c590de7481098df279ed860336f"
format_stamp: "Formatted at 2026-06-19 13:14:28 on dist-test-slave-mkt3"
server_key: "de03228dc28b325da8b71d960d942425"
server_key_iv: "206fff252983f3cd158cadf19fda2ce0"
server_key_version: "encryptionkey@0"
I20260619 13:14:28.355875 8307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:14:28.359346 8322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:28.360664 8307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 13:14:28.360813 8307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "745e8c590de7481098df279ed860336f"
format_stamp: "Formatted at 2026-06-19 13:14:28 on dist-test-slave-mkt3"
server_key: "de03228dc28b325da8b71d960d942425"
server_key_iv: "206fff252983f3cd158cadf19fda2ce0"
server_key_version: "encryptionkey@0"
I20260619 13:14:28.361013 8307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:28.380079 8307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:28.384163 8307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:28.384382 8307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:28.386983 8307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:28.387084 8307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:28.387226 8307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:28.387297 8307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:14:28 dist-test-slave-mkt3 krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874868, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:14:28.409235 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.388060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40309 (local address 127.0.97.3:49039)
0619 13:14:28.388934 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:28.388971 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:28.390865 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:28.391553 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:28.391561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:28.392264 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:28.393072 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:28.393101 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.394467 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.394474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:28.395771 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:28.395782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.396645 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.396656 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:28.396989 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.397701 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:28.397722 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:28.400552 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:28.404276 (+ 3724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:28.404290 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:28.404308 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:28.404927 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:28.405293 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:28.405299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:28.405303 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:28.405611 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:28.406279 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:28.406288 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:28.406693 (+ 405us) client_negotiation.cc:769] Sending connection context
0619 13:14:28.407056 (+ 363us) client_negotiation.cc:240] Negotiation successful
0619 13:14:28.407638 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":461,"threads_started":1}
I20260619 13:14:28.410167 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.387396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49039 (local address 127.0.97.62:40309)
0619 13:14:28.387563 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:28.387570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:28.389093 (+ 1523us) server_negotiation.cc:407] Connection header received
0619 13:14:28.391146 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:28.391152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:28.391230 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:28.391320 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:28.393252 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.394294 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.395890 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.396406 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.396526 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.400821 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:28.400828 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:28.400832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:28.400870 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:28.404120 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:28.405051 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:28.405057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:28.405061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:28.405187 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:28.405754 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:28.405759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:28.405762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:28.406067 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:28.406198 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:28.409743 (+ 3545us) server_negotiation.cc:299] Negotiation successful
0619 13:14:28.409947 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:14:28.427582 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.415260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40309 (local address 127.0.97.3:41333)
0619 13:14:28.415861 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:28.415880 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:28.416006 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:28.416297 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:28.416301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:28.416765 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:28.417033 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:28.417042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.418108 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.418114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:28.419081 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:28.419093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.419788 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.419794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:28.419958 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.420621 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:28.420637 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:28.421672 (+ 1035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:28.424710 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:28.424716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:28.424720 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:28.425110 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:28.425610 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:28.425613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:28.425615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:28.425740 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:28.426361 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:28.426370 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:28.426591 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 13:14:28.426766 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 13:14:28.427110 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":341,"threads_started":1}
I20260619 13:14:28.427793 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.415786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41333 (local address 127.0.97.62:40309)
0619 13:14:28.415944 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:28.415950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:28.415967 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:14:28.416122 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:28.416126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:28.416180 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:28.416260 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:28.417169 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.417992 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.419207 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.419621 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.419727 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.421929 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:28.421935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:28.421938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:28.421969 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:28.424579 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:28.425266 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:28.425273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:28.425278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:28.425489 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:28.425885 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:28.425889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:28.425891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:28.426240 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:28.426398 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:28.427155 (+ 757us) server_negotiation.cc:299] Negotiation successful
0619 13:14:28.427394 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:14:28.441911 8164 heartbeater.cc:503] Master 127.0.97.62:40309 was elected leader, sending a full tablet report...
I20260619 13:14:28.443116 8441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34737 every 8 connection(s)
I20260619 13:14:28.443111 8307 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34737
I20260619 13:14:28.444378 8307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:14:28.445727 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 8307
I20260619 13:14:28.445909 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:14:28.446347 388 external_mini_cluster.cc:1428] Setting key f42908a7e8a11877829d37bc27be0e0f
I20260619 13:14:28.446197 8442 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40309
I20260619 13:14:28.446949 8442 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:28.447633 8442 heartbeater.cc:511] Master 127.0.97.62:40309 requested a full tablet report, sending...
I20260619 13:14:28.448910 7958 ts_manager.cc:194] Registered new tserver with Master: 745e8c590de7481098df279ed860336f (127.0.97.3:34737)
I20260619 13:14:28.449635 7958 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49039
I20260619 13:14:28.460898 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 13:14:28 dist-test-slave-mkt3 krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874868, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 13:14:28 dist-test-slave-mkt3 krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:28 dist-test-slave-mkt3 krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 13:14:28.502498 8006 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:28.493140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.0.97.62:40309)
0619 13:14:28.493287 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:28.493291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:28.493565 (+ 274us) server_negotiation.cc:407] Connection header received
0619 13:14:28.493754 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:28.493764 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:28.493860 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:28.493946 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:28.495211 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.495932 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.497052 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.497442 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.497531 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.502298 (+ 4767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39432: BlockingRecv error: recv got EOF from 127.0.0.1:39432 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 13:14:28 dist-test-slave-mkt3 krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874868, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:14:28.517071 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.505197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.0.97.62:40309)
0619 13:14:28.505341 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:28.505346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:28.505593 (+ 247us) server_negotiation.cc:407] Connection header received
0619 13:14:28.505718 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:28.505721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:28.505798 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:28.505912 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:28.506946 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.507630 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.508796 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.509256 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.509337 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.512560 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:28.512565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:28.512568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:28.512597 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:28.514929 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:28.515585 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:28.515588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:28.515591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:28.515692 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:28.515991 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:28.515994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:28.515996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:28.516249 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:28.516419 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:28.516648 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 13:14:28.516817 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:14:28.520475 7958 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:14:28.527244 7958 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:14:28.546500 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.537995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44349 (local address 127.0.0.1:50780)
0619 13:14:28.538751 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:28.538792 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:28.538951 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:28.540014 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:28.540020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:28.540049 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:28.540408 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:28.540420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.542347 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.542352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:28.544223 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:28.544234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.545591 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.545597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:28.545801 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.545891 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 13:14:28.545992 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 13:14:28.546115 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":1,"thread_start_us":1460,"threads_started":1}
I20260619 13:14:28.547016 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.538473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.0.97.1:44349)
0619 13:14:28.539417 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:28.539426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:28.539455 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:14:28.539546 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:28.539553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:28.539806 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:28.540574 (+ 768us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:28.540775 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.542211 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.544355 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.545391 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.546381 (+ 990us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.546579 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:28.546645 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 13:14:28.546814 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":688,"threads_started":1}
I20260619 13:14:28.547670 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.538069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36587 (local address 127.0.0.1:33718)
0619 13:14:28.539675 (+ 1606us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:28.539690 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:28.539790 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:28.540924 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:28.540928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:28.540951 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:28.541317 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:28.541329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.543087 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.543093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:28.545208 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:28.545220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.546924 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.546929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:28.547088 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.547122 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 13:14:28.547179 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 13:14:28.547301 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1506,"mutex_wait_us":19,"thread_start_us":520,"threads_started":1}
I20260619 13:14:28.548382 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.538300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.0.97.2:36587)
0619 13:14:28.540228 (+ 1928us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:28.540246 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:28.540271 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:14:28.540355 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:28.540362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:28.540574 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:28.540736 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:28.541756 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.542963 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.545405 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.546718 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.547869 (+ 1151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.548018 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:28.548072 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 13:14:28.548174 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1789,"thread_start_us":1672,"threads_started":1}
I20260619 13:14:28.550493 8094 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2386fa3a624bc19a13b38a8a640df2 (DEFAULT_TABLE table=test-table [id=03433311138f4ca085bfa825fdd912b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:28.551862 8094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2386fa3a624bc19a13b38a8a640df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:28.551620 8238 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2386fa3a624bc19a13b38a8a640df2 (DEFAULT_TABLE table=test-table [id=03433311138f4ca085bfa825fdd912b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:28.552892 8238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2386fa3a624bc19a13b38a8a640df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:28.556058 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.540276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.0.97.3:34737)
0619 13:14:28.540851 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:28.540859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:28.542022 (+ 1163us) server_negotiation.cc:407] Connection header received
0619 13:14:28.542111 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:28.542117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:28.542318 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:28.542480 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:28.543404 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.544974 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.547632 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.555085 (+ 7453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.555206 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.555441 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:28.555629 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 13:14:28.555770 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":253,"threads_started":1}
I20260619 13:14:28.556071 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.539824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34737 (local address 127.0.0.1:48322)
0619 13:14:28.541774 (+ 1950us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:28.541789 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:28.541913 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:28.542733 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:28.542737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:28.542759 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:28.543166 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:28.543184 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.545141 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.545147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:28.547471 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:28.547483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.555286 (+ 7803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.555292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:28.555457 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.555490 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:14:28.555570 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 13:14:28.555703 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1845,"mutex_wait_us":3,"thread_start_us":1730,"threads_started":1}
I20260619 13:14:28.558995 8372 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2386fa3a624bc19a13b38a8a640df2 (DEFAULT_TABLE table=test-table [id=03433311138f4ca085bfa825fdd912b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:28.560199 8372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2386fa3a624bc19a13b38a8a640df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:28.565769 8466 tablet_bootstrap.cc:492] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04: Bootstrap starting.
I20260619 13:14:28.567541 8467 tablet_bootstrap.cc:492] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883: Bootstrap starting.
I20260619 13:14:28.570070 8467 tablet_bootstrap.cc:654] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:28.570266 8466 tablet_bootstrap.cc:654] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:28.571022 8467 log.cc:824] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:28.571415 8466 log.cc:824] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:28.572157 8469 tablet_bootstrap.cc:492] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f: Bootstrap starting.
I20260619 13:14:28.573880 8467 tablet_bootstrap.cc:492] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883: No bootstrap required, opened a new log
I20260619 13:14:28.574116 8467 ts_tablet_manager.cc:1397] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260619 13:14:28.575316 8469 tablet_bootstrap.cc:654] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:28.576447 8466 tablet_bootstrap.cc:492] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04: No bootstrap required, opened a new log
I20260619 13:14:28.576416 8469 log.cc:824] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:28.576639 8466 ts_tablet_manager.cc:1397] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260619 13:14:28.579241 8469 tablet_bootstrap.cc:492] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f: No bootstrap required, opened a new log
I20260619 13:14:28.579486 8469 ts_tablet_manager.cc:1397] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:14:28.583441 8466 raft_consensus.cc:348] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:28.583796 8466 raft_consensus.cc:374] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:28.583885 8466 raft_consensus.cc:729] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8178fb2a7140ca9a532e3ebb39dd04, State: Initialized, Role: FOLLOWER
I20260619 13:14:28.583621 8467 raft_consensus.cc:348] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:28.584009 8467 raft_consensus.cc:374] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:28.584108 8467 raft_consensus.cc:729] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33f57b01113407a84cc1b6bd8a78883, State: Initialized, Role: FOLLOWER
I20260619 13:14:28.584549 8466 consensus_queue.cc:260] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:28.584722 8467 consensus_queue.cc:260] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:28.586223 8466 ts_tablet_manager.cc:1428] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20260619 13:14:28.588009 8303 heartbeater.cc:503] Master 127.0.97.62:40309 was elected leader, sending a full tablet report...
I20260619 13:14:28.588568 8467 ts_tablet_manager.cc:1428] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260619 13:14:28.589293 8469 raft_consensus.cc:348] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:28.589672 8469 raft_consensus.cc:374] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:28.589772 8469 raft_consensus.cc:729] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745e8c590de7481098df279ed860336f, State: Initialized, Role: FOLLOWER
I20260619 13:14:28.590376 8469 consensus_queue.cc:260] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:28.591490 8442 heartbeater.cc:503] Master 127.0.97.62:40309 was elected leader, sending a full tablet report...
I20260619 13:14:28.591939 8469 ts_tablet_manager.cc:1428] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f: Time spent starting tablet: real 0.012s user 0.006s sys 0.008s
W20260619 13:14:28.685536 8165 tablet.cc:2369] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:14:28.695725 8443 tablet.cc:2369] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:14:28.701763 8304 tablet.cc:2369] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:14:28.842391 8473 raft_consensus.cc:482] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:14:28.842695 8473 raft_consensus.cc:504] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:28.844336 8473 leader_election.cc:288] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf8178fb2a7140ca9a532e3ebb39dd04 (127.0.97.1:44349), 745e8c590de7481098df279ed860336f (127.0.97.3:34737)
I20260619 13:14:28.851009 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.844793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55615 (local address 127.0.97.1:44349)
0619 13:14:28.845024 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:28.845031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:28.845460 (+ 429us) server_negotiation.cc:407] Connection header received
0619 13:14:28.845640 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:28.845646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:28.845741 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:28.845827 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:28.846496 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.847401 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.849247 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.850095 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.850191 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.850264 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:28.850643 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 13:14:28.850781 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:14:28.851089 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.844691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44349 (local address 127.0.97.2:55615)
0619 13:14:28.845359 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:28.845375 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:28.845513 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:28.845953 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:28.845956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:28.845981 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:28.846332 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:28.846344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.847528 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.847534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:28.849093 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:28.849105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.850313 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.850320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:28.850481 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.850517 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 13:14:28.850607 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 13:14:28.850700 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":20,"thread_start_us":356,"threads_started":1}
I20260619 13:14:28.851195 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.845676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56697 (local address 127.0.97.3:34737)
0619 13:14:28.845905 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:28.845910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:28.846131 (+ 221us) server_negotiation.cc:407] Connection header received
0619 13:14:28.846266 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:28.846272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:28.846344 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:28.846439 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:28.846992 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.847792 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.849532 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.850378 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.850492 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.850582 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:28.850861 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 13:14:28.850975 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:14:28.851508 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.845455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34737 (local address 127.0.97.2:56697)
0619 13:14:28.846043 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:28.846057 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:28.846157 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:28.846496 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:28.846501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:28.846533 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:28.846860 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:28.846872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.847929 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.847933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:28.849384 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:28.849398 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.850558 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.850562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:28.850709 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.850741 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 13:14:28.850832 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 13:14:28.850952 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":20,"thread_start_us":261,"threads_started":1}
I20260619 13:14:28.852051 8114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2386fa3a624bc19a13b38a8a640df2" candidate_uuid: "f33f57b01113407a84cc1b6bd8a78883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" is_pre_election: true
I20260619 13:14:28.852317 8397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2386fa3a624bc19a13b38a8a640df2" candidate_uuid: "f33f57b01113407a84cc1b6bd8a78883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745e8c590de7481098df279ed860336f" is_pre_election: true
I20260619 13:14:28.852516 8114 raft_consensus.cc:2454] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f33f57b01113407a84cc1b6bd8a78883 in term 0.
I20260619 13:14:28.852766 8397 raft_consensus.cc:2454] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f33f57b01113407a84cc1b6bd8a78883 in term 0.
I20260619 13:14:28.853086 8186 leader_election.cc:302] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf8178fb2a7140ca9a532e3ebb39dd04, f33f57b01113407a84cc1b6bd8a78883; no voters:
I20260619 13:14:28.853390 8473 raft_consensus.cc:2781] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:14:28.853513 8473 raft_consensus.cc:482] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:14:28.853605 8473 raft_consensus.cc:3037] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:28.855703 8473 raft_consensus.cc:504] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:28.856211 8473 leader_election.cc:288] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [CANDIDATE]: Term 1 election: Requested vote from peers cf8178fb2a7140ca9a532e3ebb39dd04 (127.0.97.1:44349), 745e8c590de7481098df279ed860336f (127.0.97.3:34737)
I20260619 13:14:28.856612 8114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2386fa3a624bc19a13b38a8a640df2" candidate_uuid: "f33f57b01113407a84cc1b6bd8a78883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04"
I20260619 13:14:28.856662 8397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2386fa3a624bc19a13b38a8a640df2" candidate_uuid: "f33f57b01113407a84cc1b6bd8a78883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745e8c590de7481098df279ed860336f"
I20260619 13:14:28.856769 8114 raft_consensus.cc:3037] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:28.856838 8397 raft_consensus.cc:3037] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:28.859133 8114 raft_consensus.cc:2454] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f33f57b01113407a84cc1b6bd8a78883 in term 1.
I20260619 13:14:28.859138 8397 raft_consensus.cc:2454] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f33f57b01113407a84cc1b6bd8a78883 in term 1.
I20260619 13:14:28.859469 8186 leader_election.cc:302] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf8178fb2a7140ca9a532e3ebb39dd04, f33f57b01113407a84cc1b6bd8a78883; no voters:
I20260619 13:14:28.859692 8473 raft_consensus.cc:2781] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:28.860056 8473 raft_consensus.cc:686] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [term 1 LEADER]: Becoming Leader. State: Replica: f33f57b01113407a84cc1b6bd8a78883, State: Running, Role: LEADER
I20260619 13:14:28.860528 8473 consensus_queue.cc:237] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:28.864629 7958 catalog_manager.cc:5495] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 reported cstate change: term changed from 0 to 1, leader changed from <none> to f33f57b01113407a84cc1b6bd8a78883 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f33f57b01113407a84cc1b6bd8a78883" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } health_report { overall_health: UNKNOWN } } }
I20260619 13:14:28.910871 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.905977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.0.97.2:36587)
0619 13:14:28.906128 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:28.906133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:28.906198 (+ 65us) server_negotiation.cc:407] Connection header received
0619 13:14:28.906421 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:28.906424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:28.906508 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:28.906621 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:14:28.906898 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.907626 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.908936 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:28.909398 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:28.909489 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:28.909887 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:14:28.910437 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:14:28.910535 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:28.910614 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 13:14:28.910673 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 13:14:28.918462 8397 raft_consensus.cc:1261] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Refusing update from remote peer f33f57b01113407a84cc1b6bd8a78883: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:28.918512 8114 raft_consensus.cc:1261] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Refusing update from remote peer f33f57b01113407a84cc1b6bd8a78883: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:28.919291 8477 consensus_queue.cc:1035] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:28.919605 8473 consensus_queue.cc:1035] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:28.930929 8480 mvcc.cc:205] Tried to move back new op lower bound from 7298559463081054208 to 7298559462856302592. Current Snapshot: MvccSnapshot[applied={T|T < 7298559463081054208 or (T in {7298559463081054208})}]
I20260619 13:14:28.931825 8481 mvcc.cc:205] Tried to move back new op lower bound from 7298559463081054208 to 7298559462856302592. Current Snapshot: MvccSnapshot[applied={T|T < 7298559463081054208}]
I20260619 13:14:28.931984 8482 mvcc.cc:205] Tried to move back new op lower bound from 7298559463081054208 to 7298559462856302592. Current Snapshot: MvccSnapshot[applied={T|T < 7298559463081054208}]
I20260619 13:14:28.940629 7957 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:28.940831 7957 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:28.943753 7957 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d: Sending DeleteTablet for 3 replicas of tablet 2b2386fa3a624bc19a13b38a8a640df2
I20260619 13:14:28.944780 8238 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b2386fa3a624bc19a13b38a8a640df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:28 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33718
I20260619 13:14:28.944754 8094 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b2386fa3a624bc19a13b38a8a640df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:28 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50780
I20260619 13:14:28.944814 8372 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b2386fa3a624bc19a13b38a8a640df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:28 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48322
I20260619 13:14:28.946033 7957 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:14:28.946816 7957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:14:28.948242 8491 tablet_replica.cc:330] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f: stopping tablet replica
I20260619 13:14:28.948671 8491 raft_consensus.cc:2229] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:28.949095 8491 raft_consensus.cc:2258] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:28.951987 8491 ts_tablet_manager.cc:1905] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:28.958606 8490 tablet_replica.cc:330] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04: stopping tablet replica
I20260619 13:14:28.959015 8490 raft_consensus.cc:2229] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:28.959461 8490 raft_consensus.cc:2258] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:28.965219 8490 ts_tablet_manager.cc:1905] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:28.967146 8238 tablet_service.cc:1467] Processing CreateTablet for tablet 2e28a5b277d743618ce19b55027e8248 (DEFAULT_TABLE table=test-table [id=ff35b60c34dc448fac8102209df3b71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:28.967710 8238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e28a5b277d743618ce19b55027e8248. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:28.967919 8094 tablet_service.cc:1467] Processing CreateTablet for tablet 2e28a5b277d743618ce19b55027e8248 (DEFAULT_TABLE table=test-table [id=ff35b60c34dc448fac8102209df3b71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:28.968298 8094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e28a5b277d743618ce19b55027e8248. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:28.949815 8489 tablet_replica.cc:330] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883: stopping tablet replica
I20260619 13:14:28.969089 8372 tablet_service.cc:1467] Processing CreateTablet for tablet 2e28a5b277d743618ce19b55027e8248 (DEFAULT_TABLE table=test-table [id=ff35b60c34dc448fac8102209df3b71d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:28.969518 8372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e28a5b277d743618ce19b55027e8248. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:28.970111 8491 ts_tablet_manager.cc:1918] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:28.970288 8491 log.cc:1196] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/2b2386fa3a624bc19a13b38a8a640df2
I20260619 13:14:28.970834 8491 ts_tablet_manager.cc:1939] T 2b2386fa3a624bc19a13b38a8a640df2 P 745e8c590de7481098df279ed860336f: Deleting consensus metadata
I20260619 13:14:28.971139 8489 raft_consensus.cc:2229] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:14:28.971793 8489 raft_consensus.cc:2258] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:28.972482 7944 catalog_manager.cc:4841] TS 745e8c590de7481098df279ed860336f (127.0.97.3:34737): tablet 2b2386fa3a624bc19a13b38a8a640df2 (table test-table [id=03433311138f4ca085bfa825fdd912b0]) successfully deleted
I20260619 13:14:28.975458 8469 tablet_bootstrap.cc:492] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f: Bootstrap starting.
I20260619 13:14:28.976099 8489 ts_tablet_manager.cc:1905] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:28.976902 8469 tablet_bootstrap.cc:654] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:28.979480 8466 tablet_bootstrap.cc:492] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04: Bootstrap starting.
I20260619 13:14:28.981272 8466 tablet_bootstrap.cc:654] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:28.983633 8490 ts_tablet_manager.cc:1918] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:28.983878 8490 log.cc:1196] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/2b2386fa3a624bc19a13b38a8a640df2
I20260619 13:14:28.984472 8490 ts_tablet_manager.cc:1939] T 2b2386fa3a624bc19a13b38a8a640df2 P cf8178fb2a7140ca9a532e3ebb39dd04: Deleting consensus metadata
I20260619 13:14:28.985337 8466 tablet_bootstrap.cc:492] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04: No bootstrap required, opened a new log
I20260619 13:14:28.985482 8466 ts_tablet_manager.cc:1397] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260619 13:14:28.986078 7944 catalog_manager.cc:4841] TS cf8178fb2a7140ca9a532e3ebb39dd04 (127.0.97.1:44349): tablet 2b2386fa3a624bc19a13b38a8a640df2 (table test-table [id=03433311138f4ca085bfa825fdd912b0]) successfully deleted
I20260619 13:14:28.986161 8469 tablet_bootstrap.cc:492] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f: No bootstrap required, opened a new log
I20260619 13:14:28.986277 8466 raft_consensus.cc:348] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:28.986415 8469 ts_tablet_manager.cc:1397] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260619 13:14:28.986495 8466 raft_consensus.cc:374] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:28.986584 8466 raft_consensus.cc:729] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8178fb2a7140ca9a532e3ebb39dd04, State: Initialized, Role: FOLLOWER
I20260619 13:14:28.986821 8466 consensus_queue.cc:260] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:28.987336 8466 ts_tablet_manager.cc:1428] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 13:14:28.987268 8469 raft_consensus.cc:348] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:28.987541 8469 raft_consensus.cc:374] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:28.988562 8469 raft_consensus.cc:729] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745e8c590de7481098df279ed860336f, State: Initialized, Role: FOLLOWER
I20260619 13:14:28.988894 8469 consensus_queue.cc:260] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:28.990022 8469 ts_tablet_manager.cc:1428] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 13:14:28.992462 8467 tablet_bootstrap.cc:492] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883: Bootstrap starting.
I20260619 13:14:28.994107 8467 tablet_bootstrap.cc:654] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:28.994590 8489 ts_tablet_manager.cc:1918] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:28.994889 8489 log.cc:1196] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/2b2386fa3a624bc19a13b38a8a640df2
I20260619 13:14:28.995436 8489 ts_tablet_manager.cc:1939] T 2b2386fa3a624bc19a13b38a8a640df2 P f33f57b01113407a84cc1b6bd8a78883: Deleting consensus metadata
I20260619 13:14:28.995951 8472 raft_consensus.cc:482] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:14:28.996126 8472 raft_consensus.cc:504] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:28.997160 8467 tablet_bootstrap.cc:492] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883: No bootstrap required, opened a new log
I20260619 13:14:28.997310 8467 ts_tablet_manager.cc:1397] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 13:14:28.997876 8472 leader_election.cc:288] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f33f57b01113407a84cc1b6bd8a78883 (127.0.97.2:36587), 745e8c590de7481098df279ed860336f (127.0.97.3:34737)
I20260619 13:14:28.998150 8467 raft_consensus.cc:348] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:28.998370 8467 raft_consensus.cc:374] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:28.998476 8467 raft_consensus.cc:729] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33f57b01113407a84cc1b6bd8a78883, State: Initialized, Role: FOLLOWER
I20260619 13:14:28.999042 8467 consensus_queue.cc:260] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.001982 8467 ts_tablet_manager.cc:1428] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 13:14:29.007143 7945 catalog_manager.cc:4841] TS f33f57b01113407a84cc1b6bd8a78883 (127.0.97.2:36587): tablet 2b2386fa3a624bc19a13b38a8a640df2 (table test-table [id=03433311138f4ca085bfa825fdd912b0]) successfully deleted
I20260619 13:14:29.010172 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.998500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36587 (local address 127.0.97.1:34207)
0619 13:14:29.002912 (+ 4412us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:29.002931 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:29.003108 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:29.003608 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:29.003613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:29.003643 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:29.004056 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.004068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.005520 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.005525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.007783 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.007799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.009397 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.009403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.009552 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.009575 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:14:29.009693 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 13:14:29.009824 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4285,"thread_start_us":9117,"threads_started":1}
I20260619 13:14:29.010159 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.998809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34207 (local address 127.0.97.2:36587)
0619 13:14:28.998968 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:28.998974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:29.003007 (+ 4033us) server_negotiation.cc:407] Connection header received
0619 13:14:29.003266 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:29.003271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:29.003344 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:29.003407 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:29.004224 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.005349 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.007991 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.009166 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.009295 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.009390 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:29.009718 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 13:14:29.009875 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:14:29.011023 8258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e28a5b277d743618ce19b55027e8248" candidate_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33f57b01113407a84cc1b6bd8a78883" is_pre_election: true
I20260619 13:14:29.011448 8258 raft_consensus.cc:2454] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf8178fb2a7140ca9a532e3ebb39dd04 in term 0.
I20260619 13:14:29.012077 8048 leader_election.cc:302] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf8178fb2a7140ca9a532e3ebb39dd04, f33f57b01113407a84cc1b6bd8a78883; no voters:
I20260619 13:14:29.013540 8472 raft_consensus.cc:2781] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:14:29.013659 8472 raft_consensus.cc:482] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:14:29.013747 8472 raft_consensus.cc:3037] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:29.014276 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.999372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58495 (local address 127.0.97.3:34737)
0619 13:14:28.999514 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:28.999519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:29.008096 (+ 8577us) server_negotiation.cc:407] Connection header received
0619 13:14:29.008255 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:29.008261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:29.008349 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:29.008434 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:29.009024 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.010147 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.012441 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.013239 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.013334 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.013397 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:29.013900 (+ 503us) server_negotiation.cc:299] Negotiation successful
0619 13:14:29.014030 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 13:14:29.014317 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:28.999007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34737 (local address 127.0.97.1:58495)
0619 13:14:29.007976 (+ 8969us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:29.007994 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:29.008113 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:29.008475 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:29.008480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:29.008507 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:29.008836 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.008847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.010318 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.010323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.012245 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.012260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.013438 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.013444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.013637 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.013675 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 13:14:29.013812 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 13:14:29.013966 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8850,"thread_start_us":3589,"threads_started":1}
I20260619 13:14:29.016435 8472 raft_consensus.cc:504] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.017391 8258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e28a5b277d743618ce19b55027e8248" candidate_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33f57b01113407a84cc1b6bd8a78883"
I20260619 13:14:29.017526 8397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e28a5b277d743618ce19b55027e8248" candidate_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745e8c590de7481098df279ed860336f" is_pre_election: true
I20260619 13:14:29.017590 8258 raft_consensus.cc:3037] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:29.017735 8397 raft_consensus.cc:2454] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf8178fb2a7140ca9a532e3ebb39dd04 in term 0.
I20260619 13:14:29.019850 8258 raft_consensus.cc:2454] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf8178fb2a7140ca9a532e3ebb39dd04 in term 1.
I20260619 13:14:29.020211 8048 leader_election.cc:302] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf8178fb2a7140ca9a532e3ebb39dd04, f33f57b01113407a84cc1b6bd8a78883; no voters:
I20260619 13:14:29.020556 8472 leader_election.cc:288] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [CANDIDATE]: Term 1 election: Requested vote from peers f33f57b01113407a84cc1b6bd8a78883 (127.0.97.2:36587), 745e8c590de7481098df279ed860336f (127.0.97.3:34737)
I20260619 13:14:29.020562 8397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e28a5b277d743618ce19b55027e8248" candidate_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745e8c590de7481098df279ed860336f"
I20260619 13:14:29.020738 8397 raft_consensus.cc:3037] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:29.020741 8472 raft_consensus.cc:2781] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:29.020866 8472 raft_consensus.cc:686] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 LEADER]: Becoming Leader. State: Replica: cf8178fb2a7140ca9a532e3ebb39dd04, State: Running, Role: LEADER
I20260619 13:14:29.021255 8472 consensus_queue.cc:237] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.023559 8397 raft_consensus.cc:2454] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf8178fb2a7140ca9a532e3ebb39dd04 in term 1.
I20260619 13:14:29.026234 7957 catalog_manager.cc:5495] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf8178fb2a7140ca9a532e3ebb39dd04 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } health_report { overall_health: UNKNOWN } } }
Jun 19 13:14:29 dist-test-slave-mkt3 krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874866, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:14:29.092742 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:29.079989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40309 (local address 127.0.0.1:39452)
0619 13:14:29.080757 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:29.080777 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:29.080919 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:29.081240 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:29.081244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:29.081967 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:29.082152 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.082161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.083123 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.083127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.084018 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.084030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.084693 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.084699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.084855 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.085516 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:29.085533 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:29.087748 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:29.090434 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:29.090445 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:29.090453 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:29.090837 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:29.091146 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:29.091151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:29.091155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:29.091306 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:29.091756 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:29.091764 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:29.091908 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 13:14:29.092051 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 13:14:29.092307 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":536,"threads_started":1}
I20260619 13:14:29.092814 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:29.080028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.0.97.62:40309)
0619 13:14:29.080611 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:29.080617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:29.080862 (+ 245us) server_negotiation.cc:407] Connection header received
0619 13:14:29.081040 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:29.081043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:29.081108 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:29.081209 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:29.082290 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.082989 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.084145 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.084550 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.084653 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.087912 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:29.087916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:29.087919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:29.087949 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:29.090326 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:29.090936 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:29.090942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:29.090944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:29.091051 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:29.091400 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:29.091405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:29.091408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:29.091638 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:29.091769 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:29.092158 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 13:14:29.092446 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":429,"thread_start_us":279,"threads_started":1}
I20260619 13:14:29.098151 7957 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39452:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 13:14:29.099149 7957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:14:29.105669 8238 tablet_service.cc:1467] Processing CreateTablet for tablet bd3682cb2d2845428c53a9f7d7b513d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ee5cdc3d19fd4f29a4b3b8184c3c9444]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:14:29.105634 8094 tablet_service.cc:1467] Processing CreateTablet for tablet bd3682cb2d2845428c53a9f7d7b513d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ee5cdc3d19fd4f29a4b3b8184c3c9444]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:14:29.106252 8238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3682cb2d2845428c53a9f7d7b513d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:29.106477 8094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3682cb2d2845428c53a9f7d7b513d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:29.106644 8372 tablet_service.cc:1467] Processing CreateTablet for tablet bd3682cb2d2845428c53a9f7d7b513d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ee5cdc3d19fd4f29a4b3b8184c3c9444]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 13:14:29.107110 8372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd3682cb2d2845428c53a9f7d7b513d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:29.113361 8466 tablet_bootstrap.cc:492] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04: Bootstrap starting.
I20260619 13:14:29.113523 8467 tablet_bootstrap.cc:492] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883: Bootstrap starting.
I20260619 13:14:29.113612 8469 tablet_bootstrap.cc:492] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f: Bootstrap starting.
I20260619 13:14:29.115007 8466 tablet_bootstrap.cc:654] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:29.115115 8469 tablet_bootstrap.cc:654] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:29.115120 8467 tablet_bootstrap.cc:654] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:29.127332 8466 tablet_bootstrap.cc:492] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04: No bootstrap required, opened a new log
I20260619 13:14:29.127544 8466 ts_tablet_manager.cc:1397] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20260619 13:14:29.128216 8469 tablet_bootstrap.cc:492] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f: No bootstrap required, opened a new log
I20260619 13:14:29.128223 8467 tablet_bootstrap.cc:492] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883: No bootstrap required, opened a new log
I20260619 13:14:29.128374 8469 ts_tablet_manager.cc:1397] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260619 13:14:29.128381 8467 ts_tablet_manager.cc:1397] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.011s
I20260619 13:14:29.129076 8469 raft_consensus.cc:348] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.129128 8467 raft_consensus.cc:348] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.129314 8469 raft_consensus.cc:374] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:29.129338 8467 raft_consensus.cc:374] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:29.129390 8469 raft_consensus.cc:729] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745e8c590de7481098df279ed860336f, State: Initialized, Role: FOLLOWER
I20260619 13:14:29.129406 8467 raft_consensus.cc:729] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33f57b01113407a84cc1b6bd8a78883, State: Initialized, Role: FOLLOWER
I20260619 13:14:29.129648 8469 consensus_queue.cc:260] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.129667 8467 consensus_queue.cc:260] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.130012 8466 raft_consensus.cc:348] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.130096 8473 tablet_replica.cc:439] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } } }
I20260619 13:14:29.130101 8474 tablet_replica.cc:439] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } } }
I20260619 13:14:29.130373 8474 tablet_replica.cc:442] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:14:29.130383 8473 tablet_replica.cc:442] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:14:29.130293 8466 raft_consensus.cc:374] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:29.130527 8466 raft_consensus.cc:729] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8178fb2a7140ca9a532e3ebb39dd04, State: Initialized, Role: FOLLOWER
I20260619 13:14:29.130807 8466 consensus_queue.cc:260] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.131448 8466 ts_tablet_manager.cc:1428] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 13:14:29.131330 8497 tablet_replica.cc:439] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } } }
I20260619 13:14:29.131717 8497 tablet_replica.cc:442] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:14:29.132110 8469 ts_tablet_manager.cc:1428] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 13:14:29.134761 8467 ts_tablet_manager.cc:1428] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260619 13:14:29.310679 8473 raft_consensus.cc:482] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:14:29.310899 8473 raft_consensus.cc:504] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.311668 8473 leader_election.cc:288] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf8178fb2a7140ca9a532e3ebb39dd04 (127.0.97.1:44349), 745e8c590de7481098df279ed860336f (127.0.97.3:34737)
I20260619 13:14:29.312155 8114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3682cb2d2845428c53a9f7d7b513d9" candidate_uuid: "f33f57b01113407a84cc1b6bd8a78883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" is_pre_election: true
I20260619 13:14:29.312376 8114 raft_consensus.cc:2454] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f33f57b01113407a84cc1b6bd8a78883 in term 0.
I20260619 13:14:29.312701 8397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3682cb2d2845428c53a9f7d7b513d9" candidate_uuid: "f33f57b01113407a84cc1b6bd8a78883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745e8c590de7481098df279ed860336f" is_pre_election: true
I20260619 13:14:29.312845 8186 leader_election.cc:302] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf8178fb2a7140ca9a532e3ebb39dd04, f33f57b01113407a84cc1b6bd8a78883; no voters:
I20260619 13:14:29.312917 8397 raft_consensus.cc:2454] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f33f57b01113407a84cc1b6bd8a78883 in term 0.
I20260619 13:14:29.313151 8473 raft_consensus.cc:2781] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:14:29.313282 8473 raft_consensus.cc:482] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:14:29.313396 8473 raft_consensus.cc:3037] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:29.316551 8473 raft_consensus.cc:504] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.317176 8473 leader_election.cc:288] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [CANDIDATE]: Term 1 election: Requested vote from peers cf8178fb2a7140ca9a532e3ebb39dd04 (127.0.97.1:44349), 745e8c590de7481098df279ed860336f (127.0.97.3:34737)
I20260619 13:14:29.317616 8114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3682cb2d2845428c53a9f7d7b513d9" candidate_uuid: "f33f57b01113407a84cc1b6bd8a78883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04"
I20260619 13:14:29.317689 8397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd3682cb2d2845428c53a9f7d7b513d9" candidate_uuid: "f33f57b01113407a84cc1b6bd8a78883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745e8c590de7481098df279ed860336f"
I20260619 13:14:29.317827 8114 raft_consensus.cc:3037] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:29.317924 8397 raft_consensus.cc:3037] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:29.319876 8397 raft_consensus.cc:2454] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f33f57b01113407a84cc1b6bd8a78883 in term 1.
I20260619 13:14:29.320256 8186 leader_election.cc:302] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745e8c590de7481098df279ed860336f, f33f57b01113407a84cc1b6bd8a78883; no voters:
I20260619 13:14:29.320540 8473 raft_consensus.cc:2781] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:29.320711 8114 raft_consensus.cc:2454] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f33f57b01113407a84cc1b6bd8a78883 in term 1.
I20260619 13:14:29.320698 8473 raft_consensus.cc:686] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [term 1 LEADER]: Becoming Leader. State: Replica: f33f57b01113407a84cc1b6bd8a78883, State: Running, Role: LEADER
I20260619 13:14:29.321210 8473 consensus_queue.cc:237] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.322902 8477 tablet_replica.cc:439] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883: TxnStatusTablet state changed. Reason: New leader f33f57b01113407a84cc1b6bd8a78883. Latest consensus state: current_term: 1 leader_uuid: "f33f57b01113407a84cc1b6bd8a78883" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } } }
I20260619 13:14:29.323263 8477 tablet_replica.cc:442] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:14:29.323943 8510 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 13:14:29.324155 8510 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 13:14:29.324751 7956 catalog_manager.cc:5495] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 reported cstate change: term changed from 0 to 1, leader changed from <none> to f33f57b01113407a84cc1b6bd8a78883 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f33f57b01113407a84cc1b6bd8a78883" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } health_report { overall_health: UNKNOWN } } }
I20260619 13:14:29.370075 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:29.364675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.0.97.2:36587)
0619 13:14:29.364871 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:29.364880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:29.364975 (+ 95us) server_negotiation.cc:407] Connection header received
0619 13:14:29.365235 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:29.365241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:29.365319 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:29.365414 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:14:29.365861 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.366615 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.368432 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.368849 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.368937 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.369449 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:14:29.369548 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:14:29.369606 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:29.369741 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 13:14:29.369806 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:14:29.370489 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:29.364619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36587 (local address 127.0.0.1:33732)
0619 13:14:29.364874 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:29.364896 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:29.365045 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:29.365487 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:29.365493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:29.365507 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:14:29.365697 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.365707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.366726 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.366732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.368293 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.368307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.368974 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.368980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.369193 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.369285 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:14:29.369621 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:14:29.369635 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 13:14:29.369708 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 13:14:29.369887 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 13:14:29.373611 8397 raft_consensus.cc:1261] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Refusing update from remote peer f33f57b01113407a84cc1b6bd8a78883: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:29.373678 8114 raft_consensus.cc:1261] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Refusing update from remote peer f33f57b01113407a84cc1b6bd8a78883: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:29.374176 8477 consensus_queue.cc:1035] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:29.374457 8473 consensus_queue.cc:1035] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:29.377512 8497 tablet_replica.cc:439] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04: TxnStatusTablet state changed. Reason: New leader f33f57b01113407a84cc1b6bd8a78883. Latest consensus state: current_term: 1 leader_uuid: "f33f57b01113407a84cc1b6bd8a78883" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } } }
I20260619 13:14:29.377765 8497 tablet_replica.cc:442] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:14:29.379987 8477 tablet_replica.cc:439] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f33f57b01113407a84cc1b6bd8a78883" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } } }
I20260619 13:14:29.380210 8477 tablet_replica.cc:442] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:14:29.380597 8497 tablet_replica.cc:439] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f33f57b01113407a84cc1b6bd8a78883" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } } }
I20260619 13:14:29.380811 8497 tablet_replica.cc:442] T bd3682cb2d2845428c53a9f7d7b513d9 P cf8178fb2a7140ca9a532e3ebb39dd04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:14:29.386646 8474 tablet_replica.cc:439] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f: TxnStatusTablet state changed. Reason: New leader f33f57b01113407a84cc1b6bd8a78883. Latest consensus state: current_term: 1 leader_uuid: "f33f57b01113407a84cc1b6bd8a78883" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } } }
I20260619 13:14:29.386897 8474 tablet_replica.cc:442] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:14:29.389127 8477 tablet_replica.cc:439] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f33f57b01113407a84cc1b6bd8a78883" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } } }
I20260619 13:14:29.389396 8477 tablet_replica.cc:442] T bd3682cb2d2845428c53a9f7d7b513d9 P f33f57b01113407a84cc1b6bd8a78883: This TxnStatusTablet replica's current role is: LEADER
I20260619 13:14:29.391568 8474 tablet_replica.cc:439] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f33f57b01113407a84cc1b6bd8a78883" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } } }
I20260619 13:14:29.391786 8474 tablet_replica.cc:442] T bd3682cb2d2845428c53a9f7d7b513d9 P 745e8c590de7481098df279ed860336f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 13:14:29.397078 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:29.387318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.0.97.1:44349)
0619 13:14:29.388290 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:29.388296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:29.388317 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:14:29.388398 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:29.388403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:29.388492 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:29.389196 (+ 704us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:14:29.389599 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.390740 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.394595 (+ 3855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.395263 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.395381 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.395787 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:14:29.396485 (+ 698us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:14:29.396624 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:29.396667 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 13:14:29.396744 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":743,"threads_started":1}
I20260619 13:14:29.412904 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:29.406931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38137 (local address 127.0.97.2:36587)
0619 13:14:29.407121 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:29.407128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:29.407638 (+ 510us) server_negotiation.cc:407] Connection header received
0619 13:14:29.407782 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:29.407787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:29.407871 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:29.407985 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:14:29.408437 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.409591 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.410935 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.411663 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.411774 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.412161 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:14:29.412319 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:14:29.412416 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:29.412532 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 13:14:29.412611 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:14:29.412923 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:29.406924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36587 (local address 127.0.97.1:38137)
0619 13:14:29.407554 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:29.407569 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:29.407677 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:29.408046 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:29.408051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:29.408064 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:14:29.408308 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.408316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.409705 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.409708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.410799 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.410810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.411836 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.411841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.411949 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.412000 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:14:29.412417 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:14:29.412428 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 13:14:29.412501 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 13:14:29.412654 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":308,"threads_started":1}
I20260619 13:14:29.421939 8397 raft_consensus.cc:1261] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Refusing update from remote peer cf8178fb2a7140ca9a532e3ebb39dd04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:29.422420 8258 raft_consensus.cc:1261] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Refusing update from remote peer cf8178fb2a7140ca9a532e3ebb39dd04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:29.422490 8472 consensus_queue.cc:1035] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:29.423095 8472 consensus_queue.cc:1035] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:29.485597 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:29.463689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47809 (local address 127.0.97.1:44349)
0619 13:14:29.463835 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:29.463841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:29.480944 (+ 17103us) server_negotiation.cc:407] Connection header received
0619 13:14:29.481136 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:29.481141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:29.481221 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:29.481304 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:14:29.481894 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.482581 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.483996 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.484384 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.484490 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.484881 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:14:29.484992 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:14:29.485063 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:29.485238 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 13:14:29.485318 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 13:14:29.486325 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:29.463287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44349 (local address 127.0.97.2:47809)
0619 13:14:29.480834 (+ 17547us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:29.480853 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:29.480989 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:29.481445 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:29.481448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:29.481457 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 13:14:29.481715 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.481725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.482710 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.482715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.483860 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.483872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.484537 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.484543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.484673 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.484726 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 13:14:29.485082 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 13:14:29.485093 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 13:14:29.485777 (+ 684us) client_negotiation.cc:240] Negotiation successful
0619 13:14:29.485948 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17362,"thread_start_us":17797,"threads_started":1}
I20260619 13:14:29.527419 7956 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:29.527690 7956 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:29.530948 7956 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d: Sending DeleteTablet for 3 replicas of tablet 2e28a5b277d743618ce19b55027e8248
I20260619 13:14:29.531646 8094 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e28a5b277d743618ce19b55027e8248 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:29 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50780
I20260619 13:14:29.531937 8238 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e28a5b277d743618ce19b55027e8248 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:29 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33718
I20260619 13:14:29.532629 8372 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e28a5b277d743618ce19b55027e8248 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:29 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48322
I20260619 13:14:29.543879 8526 tablet_replica.cc:330] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04: stopping tablet replica
I20260619 13:14:29.544258 8526 raft_consensus.cc:2229] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:14:29.544935 8526 raft_consensus.cc:2258] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:29.546206 8526 ts_tablet_manager.cc:1905] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:29.553489 8526 ts_tablet_manager.cc:1918] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:14:29.553732 8526 log.cc:1196] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/2e28a5b277d743618ce19b55027e8248
I20260619 13:14:29.554271 8526 ts_tablet_manager.cc:1939] T 2e28a5b277d743618ce19b55027e8248 P cf8178fb2a7140ca9a532e3ebb39dd04: Deleting consensus metadata
I20260619 13:14:29.555539 7944 catalog_manager.cc:4841] TS cf8178fb2a7140ca9a532e3ebb39dd04 (127.0.97.1:44349): tablet 2e28a5b277d743618ce19b55027e8248 (table test-table [id=ff35b60c34dc448fac8102209df3b71d]) successfully deleted
I20260619 13:14:29.560624 8528 tablet_replica.cc:330] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f: stopping tablet replica
I20260619 13:14:29.560902 8528 raft_consensus.cc:2229] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:29.561210 8528 raft_consensus.cc:2258] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:29.562407 8528 ts_tablet_manager.cc:1905] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:29.562983 8527 tablet_replica.cc:330] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883: stopping tablet replica
I20260619 13:14:29.563194 8527 raft_consensus.cc:2229] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:29.563599 8527 raft_consensus.cc:2258] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:29.564779 8527 ts_tablet_manager.cc:1905] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:29.572042 8527 ts_tablet_manager.cc:1918] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:14:29.572247 8527 log.cc:1196] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/2e28a5b277d743618ce19b55027e8248
I20260619 13:14:29.572785 8527 ts_tablet_manager.cc:1939] T 2e28a5b277d743618ce19b55027e8248 P f33f57b01113407a84cc1b6bd8a78883: Deleting consensus metadata
I20260619 13:14:29.574143 7945 catalog_manager.cc:4841] TS f33f57b01113407a84cc1b6bd8a78883 (127.0.97.2:36587): tablet 2e28a5b277d743618ce19b55027e8248 (table test-table [id=ff35b60c34dc448fac8102209df3b71d]) successfully deleted
I20260619 13:14:29.579385 8528 ts_tablet_manager.cc:1918] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:14:29.579602 8528 log.cc:1196] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/2e28a5b277d743618ce19b55027e8248
I20260619 13:14:29.580061 8528 ts_tablet_manager.cc:1939] T 2e28a5b277d743618ce19b55027e8248 P 745e8c590de7481098df279ed860336f: Deleting consensus metadata
I20260619 13:14:29.581229 7944 catalog_manager.cc:4841] TS 745e8c590de7481098df279ed860336f (127.0.97.3:34737): tablet 2e28a5b277d743618ce19b55027e8248 (table test-table [id=ff35b60c34dc448fac8102209df3b71d]) successfully deleted
Jun 19 13:14:29 dist-test-slave-mkt3 krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 13:14:29 dist-test-slave-mkt3 krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:29 dist-test-slave-mkt3 krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 13:14:29.634200 8535 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:29.608752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39454 (local address 127.0.97.62:40309)
0619 13:14:29.622542 (+ 13790us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:29.622559 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:29.622581 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:14:29.622642 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:29.622648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:29.622730 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:29.622848 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:29.624499 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.625618 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.627534 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.628206 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.628325 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.633911 (+ 5586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39454: BlockingRecv error: recv got EOF from 127.0.0.1:39454 (error 108)
Metrics: {"server-negotiator.queue_time_us":13658,"thread_start_us":13754,"threads_started":1}
Jun 19 13:14:29 dist-test-slave-mkt3 krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 13:14:29.656175 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:29.639705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39456 (local address 127.0.97.62:40309)
0619 13:14:29.639855 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:29.639860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:29.640302 (+ 442us) server_negotiation.cc:407] Connection header received
0619 13:14:29.640462 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:29.640467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:29.640540 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:29.640637 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:29.642061 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.643080 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.644869 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.645480 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.645592 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.650174 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:29.650182 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:29.650188 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:29.650227 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:29.653406 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:29.654176 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:29.654181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:29.654186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:29.654336 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:29.654749 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:29.654754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:29.654758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:29.655078 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:29.655309 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:29.655607 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 13:14:29.655820 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:14:29.659765 7956 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:14:29.660493 7956 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:14:29.667590 8094 tablet_service.cc:1467] Processing CreateTablet for tablet 0d516c6174c64665b8ae9ab2d7e1249b (DEFAULT_TABLE table=test-table [id=90f72dbe6c064627a5d969247581e03c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:29.668021 8094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d516c6174c64665b8ae9ab2d7e1249b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:29.670225 8238 tablet_service.cc:1467] Processing CreateTablet for tablet 0d516c6174c64665b8ae9ab2d7e1249b (DEFAULT_TABLE table=test-table [id=90f72dbe6c064627a5d969247581e03c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:29.670246 8372 tablet_service.cc:1467] Processing CreateTablet for tablet 0d516c6174c64665b8ae9ab2d7e1249b (DEFAULT_TABLE table=test-table [id=90f72dbe6c064627a5d969247581e03c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:29.670643 8238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d516c6174c64665b8ae9ab2d7e1249b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:29.670650 8372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d516c6174c64665b8ae9ab2d7e1249b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:29.692760 8544 tablet_bootstrap.cc:492] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04: Bootstrap starting.
I20260619 13:14:29.694304 8542 tablet_bootstrap.cc:492] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f: Bootstrap starting.
I20260619 13:14:29.696146 8543 tablet_bootstrap.cc:492] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883: Bootstrap starting.
I20260619 13:14:29.697194 8544 tablet_bootstrap.cc:654] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:29.698771 8542 tablet_bootstrap.cc:654] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:29.700124 8543 tablet_bootstrap.cc:654] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:29.710206 8543 tablet_bootstrap.cc:492] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883: No bootstrap required, opened a new log
I20260619 13:14:29.710642 8543 ts_tablet_manager.cc:1397] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.002s
I20260619 13:14:29.711527 8543 raft_consensus.cc:348] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.711974 8543 raft_consensus.cc:374] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:29.712132 8543 raft_consensus.cc:729] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33f57b01113407a84cc1b6bd8a78883, State: Initialized, Role: FOLLOWER
I20260619 13:14:29.712568 8543 consensus_queue.cc:260] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.713938 8542 tablet_bootstrap.cc:492] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f: No bootstrap required, opened a new log
I20260619 13:14:29.714131 8542 ts_tablet_manager.cc:1397] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20260619 13:14:29.714965 8542 raft_consensus.cc:348] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.715256 8542 raft_consensus.cc:374] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:29.715476 8542 raft_consensus.cc:729] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745e8c590de7481098df279ed860336f, State: Initialized, Role: FOLLOWER
I20260619 13:14:29.715790 8542 consensus_queue.cc:260] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.718822 8542 ts_tablet_manager.cc:1428] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 13:14:29.723376 8543 ts_tablet_manager.cc:1428] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20260619 13:14:29.727121 8544 tablet_bootstrap.cc:492] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04: No bootstrap required, opened a new log
I20260619 13:14:29.727285 8544 ts_tablet_manager.cc:1397] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04: Time spent bootstrapping tablet: real 0.035s user 0.020s sys 0.007s
I20260619 13:14:29.727998 8544 raft_consensus.cc:348] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.728267 8544 raft_consensus.cc:374] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:29.728415 8544 raft_consensus.cc:729] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8178fb2a7140ca9a532e3ebb39dd04, State: Initialized, Role: FOLLOWER
I20260619 13:14:29.728672 8544 consensus_queue.cc:260] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.729815 8544 ts_tablet_manager.cc:1428] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 13:14:29.869493 8474 raft_consensus.cc:482] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:14:29.869686 8474 raft_consensus.cc:504] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.871407 8474 leader_election.cc:288] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf8178fb2a7140ca9a532e3ebb39dd04 (127.0.97.1:44349), f33f57b01113407a84cc1b6bd8a78883 (127.0.97.2:36587)
I20260619 13:14:29.891685 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:29.872130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38183 (local address 127.0.97.1:44349)
0619 13:14:29.872324 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:29.872330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:29.884204 (+ 11874us) server_negotiation.cc:407] Connection header received
0619 13:14:29.884389 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:29.884395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:29.884502 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:29.884616 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:29.885492 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.886620 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.889207 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.890471 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.890569 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.890671 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:29.891241 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 13:14:29.891384 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:14:29.891789 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:29.872196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33953 (local address 127.0.97.2:36587)
0619 13:14:29.872378 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:29.872384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:29.884947 (+ 12563us) server_negotiation.cc:407] Connection header received
0619 13:14:29.885209 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:29.885214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:29.885290 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:29.885342 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:14:29.886052 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.887109 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.889537 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.890736 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.890823 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.890917 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:29.891421 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 13:14:29.891554 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 13:14:29.892532 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:29.871700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44349 (local address 127.0.97.3:38183)
0619 13:14:29.884091 (+ 12391us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:29.884111 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:29.884259 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:29.884939 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:29.884944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:29.884972 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:29.885308 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.885322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.886794 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.886798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.889019 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.889033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.890818 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.890823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.890985 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.891088 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 13:14:29.891948 (+ 860us) client_negotiation.cc:240] Negotiation successful
0619 13:14:29.892092 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12257,"mutex_wait_us":67,"thread_start_us":13011,"threads_started":1}
I20260619 13:14:29.892931 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:29.871863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36587 (local address 127.0.97.3:33953)
0619 13:14:29.884848 (+ 12985us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:29.884867 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:29.885068 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:29.885512 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:29.885516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:29.885537 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:14:29.885831 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.885876 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.887300 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.887305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.889375 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:29.889388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.891075 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.891081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:29.891262 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.891293 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:14:29.892263 (+ 970us) client_negotiation.cc:240] Negotiation successful
0619 13:14:29.892391 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12864,"mutex_wait_us":46,"spinlock_wait_cycles":30592,"thread_start_us":12470,"threads_started":1}
I20260619 13:14:29.893258 8114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d516c6174c64665b8ae9ab2d7e1249b" candidate_uuid: "745e8c590de7481098df279ed860336f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" is_pre_election: true
I20260619 13:14:29.893525 8114 raft_consensus.cc:2454] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745e8c590de7481098df279ed860336f in term 0.
I20260619 13:14:29.893545 8257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d516c6174c64665b8ae9ab2d7e1249b" candidate_uuid: "745e8c590de7481098df279ed860336f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33f57b01113407a84cc1b6bd8a78883" is_pre_election: true
I20260619 13:14:29.893755 8257 raft_consensus.cc:2454] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745e8c590de7481098df279ed860336f in term 0.
I20260619 13:14:29.894289 8326 leader_election.cc:302] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745e8c590de7481098df279ed860336f, f33f57b01113407a84cc1b6bd8a78883; no voters:
I20260619 13:14:29.894670 8474 raft_consensus.cc:2781] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:14:29.894775 8474 raft_consensus.cc:482] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:14:29.894851 8474 raft_consensus.cc:3037] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:29.897756 8474 raft_consensus.cc:504] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.898540 8474 leader_election.cc:288] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [CANDIDATE]: Term 1 election: Requested vote from peers cf8178fb2a7140ca9a532e3ebb39dd04 (127.0.97.1:44349), f33f57b01113407a84cc1b6bd8a78883 (127.0.97.2:36587)
I20260619 13:14:29.898952 8114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d516c6174c64665b8ae9ab2d7e1249b" candidate_uuid: "745e8c590de7481098df279ed860336f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04"
I20260619 13:14:29.899108 8257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d516c6174c64665b8ae9ab2d7e1249b" candidate_uuid: "745e8c590de7481098df279ed860336f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33f57b01113407a84cc1b6bd8a78883"
I20260619 13:14:29.899168 8114 raft_consensus.cc:3037] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:29.899291 8257 raft_consensus.cc:3037] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:29.902020 8257 raft_consensus.cc:2454] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745e8c590de7481098df279ed860336f in term 1.
I20260619 13:14:29.902173 8114 raft_consensus.cc:2454] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745e8c590de7481098df279ed860336f in term 1.
I20260619 13:14:29.902418 8326 leader_election.cc:302] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745e8c590de7481098df279ed860336f, f33f57b01113407a84cc1b6bd8a78883; no voters:
I20260619 13:14:29.902756 8474 raft_consensus.cc:2781] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:29.903194 8474 raft_consensus.cc:686] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [term 1 LEADER]: Becoming Leader. State: Replica: 745e8c590de7481098df279ed860336f, State: Running, Role: LEADER
I20260619 13:14:29.903586 8474 consensus_queue.cc:237] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:29.906944 7957 catalog_manager.cc:5495] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f reported cstate change: term changed from 0 to 1, leader changed from <none> to 745e8c590de7481098df279ed860336f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "745e8c590de7481098df279ed860336f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } health_report { overall_health: HEALTHY } } }
I20260619 13:14:29.920799 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:29.915395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.0.97.3:34737)
0619 13:14:29.915885 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:29.915889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:29.915905 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:14:29.915975 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:29.915978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:29.916047 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:29.916178 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:14:29.916554 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.917332 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.918776 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:29.919171 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:29.919273 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:29.919673 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:14:29.920229 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:14:29.920354 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:29.920430 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 13:14:29.920501 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":263,"threads_started":1}
I20260619 13:14:29.924351 8114 raft_consensus.cc:1261] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Refusing update from remote peer 745e8c590de7481098df279ed860336f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:29.924399 8257 raft_consensus.cc:1261] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Refusing update from remote peer 745e8c590de7481098df279ed860336f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:29.924976 8474 consensus_queue.cc:1035] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:29.925256 8550 consensus_queue.cc:1035] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:29.946882 7957 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:29.947341 7957 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:29.951244 7957 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d: Sending DeleteTablet for 3 replicas of tablet 0d516c6174c64665b8ae9ab2d7e1249b
I20260619 13:14:29.951889 8094 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d516c6174c64665b8ae9ab2d7e1249b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:29 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50780
I20260619 13:14:29.952122 8526 tablet_replica.cc:330] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04: stopping tablet replica
I20260619 13:14:29.952301 8526 raft_consensus.cc:2229] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:29.952494 8526 raft_consensus.cc:2258] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:29.952723 8238 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d516c6174c64665b8ae9ab2d7e1249b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:29 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33718
I20260619 13:14:29.952983 8527 tablet_replica.cc:330] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883: stopping tablet replica
I20260619 13:14:29.953162 8527 raft_consensus.cc:2229] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:29.953359 8527 raft_consensus.cc:2258] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:29.954419 8372 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d516c6174c64665b8ae9ab2d7e1249b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:29 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48322
I20260619 13:14:29.954640 8528 tablet_replica.cc:330] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f: stopping tablet replica
I20260619 13:14:29.954654 8527 ts_tablet_manager.cc:1905] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:29.954797 8528 raft_consensus.cc:2229] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:14:29.956202 8528 raft_consensus.cc:2258] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:29.957062 8528 ts_tablet_manager.cc:1905] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:29.960979 8527 ts_tablet_manager.cc:1918] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:29.961153 8527 log.cc:1196] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/0d516c6174c64665b8ae9ab2d7e1249b
I20260619 13:14:29.961508 8527 ts_tablet_manager.cc:1939] T 0d516c6174c64665b8ae9ab2d7e1249b P f33f57b01113407a84cc1b6bd8a78883: Deleting consensus metadata
I20260619 13:14:29.963097 7945 catalog_manager.cc:4841] TS f33f57b01113407a84cc1b6bd8a78883 (127.0.97.2:36587): tablet 0d516c6174c64665b8ae9ab2d7e1249b (table test-table [id=90f72dbe6c064627a5d969247581e03c]) successfully deleted
I20260619 13:14:29.963687 8528 ts_tablet_manager.cc:1918] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:29.963881 8528 log.cc:1196] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/0d516c6174c64665b8ae9ab2d7e1249b
I20260619 13:14:29.963881 7957 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 13:14:29.964337 8528 ts_tablet_manager.cc:1939] T 0d516c6174c64665b8ae9ab2d7e1249b P 745e8c590de7481098df279ed860336f: Deleting consensus metadata
W20260619 13:14:29.964807 7957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:14:29.965294 7944 catalog_manager.cc:4841] TS 745e8c590de7481098df279ed860336f (127.0.97.3:34737): tablet 0d516c6174c64665b8ae9ab2d7e1249b (table test-table [id=90f72dbe6c064627a5d969247581e03c]) successfully deleted
I20260619 13:14:29.966804 8526 ts_tablet_manager.cc:1905] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:29.972733 8526 ts_tablet_manager.cc:1918] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:14:29.972694 8372 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8aebbf864740a7ab2e868461a78102 (DEFAULT_TABLE table=test-table [id=cbb30df4f0394be395a347faf90ed9fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:29.972865 8526 log.cc:1196] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/0d516c6174c64665b8ae9ab2d7e1249b
I20260619 13:14:29.973130 8372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8aebbf864740a7ab2e868461a78102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:29.973228 8526 ts_tablet_manager.cc:1939] T 0d516c6174c64665b8ae9ab2d7e1249b P cf8178fb2a7140ca9a532e3ebb39dd04: Deleting consensus metadata
I20260619 13:14:29.974673 8238 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8aebbf864740a7ab2e868461a78102 (DEFAULT_TABLE table=test-table [id=cbb30df4f0394be395a347faf90ed9fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:29.975083 8238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8aebbf864740a7ab2e868461a78102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:29.981048 8543 tablet_bootstrap.cc:492] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883: Bootstrap starting.
I20260619 13:14:29.982121 8094 tablet_service.cc:1467] Processing CreateTablet for tablet 4d8aebbf864740a7ab2e868461a78102 (DEFAULT_TABLE table=test-table [id=cbb30df4f0394be395a347faf90ed9fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:14:29.982506 8094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d8aebbf864740a7ab2e868461a78102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:29.982621 8543 tablet_bootstrap.cc:654] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:29.986131 7944 catalog_manager.cc:4841] TS cf8178fb2a7140ca9a532e3ebb39dd04 (127.0.97.1:44349): tablet 0d516c6174c64665b8ae9ab2d7e1249b (table test-table [id=90f72dbe6c064627a5d969247581e03c]) successfully deleted
I20260619 13:14:29.992100 8542 tablet_bootstrap.cc:492] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f: Bootstrap starting.
I20260619 13:14:29.993876 8542 tablet_bootstrap.cc:654] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:29.996039 8544 tablet_bootstrap.cc:492] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04: Bootstrap starting.
I20260619 13:14:29.997659 8544 tablet_bootstrap.cc:654] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:30.011588 8542 tablet_bootstrap.cc:492] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f: No bootstrap required, opened a new log
I20260619 13:14:30.011828 8542 ts_tablet_manager.cc:1397] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.010s
I20260619 13:14:30.012835 8544 tablet_bootstrap.cc:492] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04: No bootstrap required, opened a new log
I20260619 13:14:30.012708 8542 raft_consensus.cc:348] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:30.012980 8542 raft_consensus.cc:374] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:30.013062 8542 raft_consensus.cc:729] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745e8c590de7481098df279ed860336f, State: Initialized, Role: FOLLOWER
I20260619 13:14:30.013197 8544 ts_tablet_manager.cc:1397] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.010s
I20260619 13:14:30.013337 8542 consensus_queue.cc:260] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:30.013877 8542 ts_tablet_manager.cc:1428] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 13:14:30.014091 8544 raft_consensus.cc:348] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:30.014382 8544 raft_consensus.cc:374] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:30.014487 8544 raft_consensus.cc:729] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8178fb2a7140ca9a532e3ebb39dd04, State: Initialized, Role: FOLLOWER
I20260619 13:14:30.018831 8543 tablet_bootstrap.cc:492] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883: No bootstrap required, opened a new log
I20260619 13:14:30.018769 8544 consensus_queue.cc:260] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:30.018986 8543 ts_tablet_manager.cc:1397] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.006s
I20260619 13:14:30.020571 8543 raft_consensus.cc:348] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:30.020833 8543 raft_consensus.cc:374] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:30.020902 8543 raft_consensus.cc:729] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33f57b01113407a84cc1b6bd8a78883, State: Initialized, Role: FOLLOWER
I20260619 13:14:30.021136 8543 consensus_queue.cc:260] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:30.021679 8543 ts_tablet_manager.cc:1428] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260619 13:14:30.022832 8544 ts_tablet_manager.cc:1428] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20260619 13:14:30.217510 8523 raft_consensus.cc:482] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:14:30.217742 8523 raft_consensus.cc:504] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:30.218423 8523 leader_election.cc:288] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f33f57b01113407a84cc1b6bd8a78883 (127.0.97.2:36587), 745e8c590de7481098df279ed860336f (127.0.97.3:34737)
I20260619 13:14:30.219022 8257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8aebbf864740a7ab2e868461a78102" candidate_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33f57b01113407a84cc1b6bd8a78883" is_pre_election: true
I20260619 13:14:30.219023 8396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8aebbf864740a7ab2e868461a78102" candidate_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745e8c590de7481098df279ed860336f" is_pre_election: true
I20260619 13:14:30.219260 8257 raft_consensus.cc:2454] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf8178fb2a7140ca9a532e3ebb39dd04 in term 0.
I20260619 13:14:30.219372 8396 raft_consensus.cc:2454] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf8178fb2a7140ca9a532e3ebb39dd04 in term 0.
I20260619 13:14:30.219762 8048 leader_election.cc:302] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf8178fb2a7140ca9a532e3ebb39dd04, f33f57b01113407a84cc1b6bd8a78883; no voters:
I20260619 13:14:30.220050 8523 raft_consensus.cc:2781] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:14:30.220177 8523 raft_consensus.cc:482] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:14:30.220256 8523 raft_consensus.cc:3037] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:30.222340 8523 raft_consensus.cc:504] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:30.222805 8523 leader_election.cc:288] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [CANDIDATE]: Term 1 election: Requested vote from peers f33f57b01113407a84cc1b6bd8a78883 (127.0.97.2:36587), 745e8c590de7481098df279ed860336f (127.0.97.3:34737)
I20260619 13:14:30.223295 8257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8aebbf864740a7ab2e868461a78102" candidate_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f33f57b01113407a84cc1b6bd8a78883"
I20260619 13:14:30.223347 8396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d8aebbf864740a7ab2e868461a78102" candidate_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "745e8c590de7481098df279ed860336f"
I20260619 13:14:30.223493 8257 raft_consensus.cc:3037] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:30.223533 8396 raft_consensus.cc:3037] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:30.226341 8257 raft_consensus.cc:2454] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf8178fb2a7140ca9a532e3ebb39dd04 in term 1.
I20260619 13:14:30.226505 8396 raft_consensus.cc:2454] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf8178fb2a7140ca9a532e3ebb39dd04 in term 1.
I20260619 13:14:30.226766 8048 leader_election.cc:302] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf8178fb2a7140ca9a532e3ebb39dd04, f33f57b01113407a84cc1b6bd8a78883; no voters:
I20260619 13:14:30.227054 8523 raft_consensus.cc:2781] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:30.227465 8523 raft_consensus.cc:686] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 LEADER]: Becoming Leader. State: Replica: cf8178fb2a7140ca9a532e3ebb39dd04, State: Running, Role: LEADER
I20260619 13:14:30.227789 8523 consensus_queue.cc:237] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } }
I20260619 13:14:30.230521 7956 catalog_manager.cc:5495] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf8178fb2a7140ca9a532e3ebb39dd04 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf8178fb2a7140ca9a532e3ebb39dd04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 } health_report { overall_health: UNKNOWN } } }
I20260619 13:14:30.257984 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:30.253444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50796 (local address 127.0.97.1:44349)
0619 13:14:30.253592 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:30.253598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:30.253692 (+ 94us) server_negotiation.cc:407] Connection header received
0619 13:14:30.253918 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:30.253923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:30.253993 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:30.254063 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:14:30.254430 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:30.255120 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:30.256361 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:30.256760 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:30.256869 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:30.257317 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:14:30.257420 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:14:30.257491 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:30.257631 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 13:14:30.257704 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 13:14:30.267122 8396 raft_consensus.cc:1261] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Refusing update from remote peer cf8178fb2a7140ca9a532e3ebb39dd04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:30.267277 8257 raft_consensus.cc:1261] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Refusing update from remote peer cf8178fb2a7140ca9a532e3ebb39dd04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:14:30.267680 8561 consensus_queue.cc:1035] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "745e8c590de7481098df279ed860336f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:30.268220 8566 consensus_queue.cc:1035] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f33f57b01113407a84cc1b6bd8a78883" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:14:30.315227 7956 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:30.315447 7956 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:14:30.318102 7956 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 924af2bae47e454798718f65b6b38e7d: Sending DeleteTablet for 3 replicas of tablet 4d8aebbf864740a7ab2e868461a78102
I20260619 13:14:30.318885 8094 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8aebbf864740a7ab2e868461a78102 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50780
I20260619 13:14:30.319023 8372 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8aebbf864740a7ab2e868461a78102 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48322
I20260619 13:14:30.319147 8526 tablet_replica.cc:330] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04: stopping tablet replica
I20260619 13:14:30.319288 8528 tablet_replica.cc:330] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f: stopping tablet replica
I20260619 13:14:30.319329 8526 raft_consensus.cc:2229] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:14:30.319326 8238 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d8aebbf864740a7ab2e868461a78102 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:14:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33718
I20260619 13:14:30.319456 8528 raft_consensus.cc:2229] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:30.319591 8527 tablet_replica.cc:330] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883: stopping tablet replica
I20260619 13:14:30.319623 8526 raft_consensus.cc:2258] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:30.319644 8528 raft_consensus.cc:2258] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:30.319748 8527 raft_consensus.cc:2229] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:14:30.319912 8527 raft_consensus.cc:2258] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:14:30.321368 8527 ts_tablet_manager.cc:1905] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:30.321391 8528 ts_tablet_manager.cc:1905] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:30.326926 8526 ts_tablet_manager.cc:1905] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:14:30.327503 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 8029
I20260619 13:14:30.328907 8527 ts_tablet_manager.cc:1918] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:14:30.329123 8527 log.cc:1196] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/4d8aebbf864740a7ab2e868461a78102
I20260619 13:14:30.329602 8527 ts_tablet_manager.cc:1939] T 4d8aebbf864740a7ab2e868461a78102 P f33f57b01113407a84cc1b6bd8a78883: Deleting consensus metadata
I20260619 13:14:30.331046 7945 catalog_manager.cc:4841] TS f33f57b01113407a84cc1b6bd8a78883 (127.0.97.2:36587): tablet 4d8aebbf864740a7ab2e868461a78102 (table test-table [id=cbb30df4f0394be395a347faf90ed9fb]) successfully deleted
I20260619 13:14:30.332528 8526 ts_tablet_manager.cc:1918] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:14:30.332640 8526 log.cc:1196] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/4d8aebbf864740a7ab2e868461a78102
I20260619 13:14:30.332940 8526 ts_tablet_manager.cc:1939] T 4d8aebbf864740a7ab2e868461a78102 P cf8178fb2a7140ca9a532e3ebb39dd04: Deleting consensus metadata
I20260619 13:14:30.333472 8528 ts_tablet_manager.cc:1918] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 13:14:30.333601 8528 log.cc:1196] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/4d8aebbf864740a7ab2e868461a78102
I20260619 13:14:30.333909 7944 catalog_manager.cc:4841] TS cf8178fb2a7140ca9a532e3ebb39dd04 (127.0.97.1:44349): tablet 4d8aebbf864740a7ab2e868461a78102 (table test-table [id=cbb30df4f0394be395a347faf90ed9fb]) successfully deleted
I20260619 13:14:30.334311 8528 ts_tablet_manager.cc:1939] T 4d8aebbf864740a7ab2e868461a78102 P 745e8c590de7481098df279ed860336f: Deleting consensus metadata
I20260619 13:14:30.335366 7944 catalog_manager.cc:4841] TS 745e8c590de7481098df279ed860336f (127.0.97.3:34737): tablet 4d8aebbf864740a7ab2e868461a78102 (table test-table [id=cbb30df4f0394be395a347faf90ed9fb]) successfully deleted
Jun 19 13:14:30 dist-test-slave-mkt3 krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260619 13:14:30.353709 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:30.328581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50800 (local address 127.0.97.1:44349)
0619 13:14:30.328721 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:30.328727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:30.334022 (+ 5295us) server_negotiation.cc:407] Connection header received
0619 13:14:30.334301 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:30.334307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:30.334372 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:30.334456 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:30.335879 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:30.336940 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:30.338131 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:30.338756 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:30.338879 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:30.347660 (+ 8781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:30.347678 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:30.347697 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:30.347749 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:30.351195 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:30.351809 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:30.351818 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:30.351822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:30.351965 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:30.352304 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:30.352309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:30.352312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:30.352719 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:30.352998 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:30.353135 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 13:14:30.353355 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 13:14:30.354368 8159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:30.473881 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 8029
I20260619 13:14:30.491583 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 8168
Jun 19 13:14:30 dist-test-slave-mkt3 krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260619 13:14:30.505944 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:30.492203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.0.97.2:36587)
0619 13:14:30.492644 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:30.492648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:30.492662 (+ 14us) server_negotiation.cc:407] Connection header received
0619 13:14:30.492711 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:30.492715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:30.492789 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:30.492875 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:30.494079 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:30.494834 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:30.496059 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:30.496436 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:30.496543 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:30.500498 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:30.500512 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:30.500528 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:30.500562 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:30.503325 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:30.503960 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:30.503968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:30.503970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:30.504101 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:30.504439 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:30.504442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:30.504444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:30.504807 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:30.505077 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:30.505380 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 13:14:30.505589 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":218,"threads_started":1}
I20260619 13:14:30.506527 8298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:30.643630 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 8168
I20260619 13:14:30.663281 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 8307
Jun 19 13:14:30 dist-test-slave-mkt3 krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260619 13:14:30.676461 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:30.663939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.0.97.3:34737)
0619 13:14:30.664383 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:30.664387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:30.664403 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:14:30.664455 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:30.664459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:30.664518 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:30.664604 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:30.665799 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:30.666503 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:30.667603 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:30.667984 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:30.668069 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:30.671840 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:30.671853 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:30.671869 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:30.671903 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:30.674323 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:30.674856 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:30.674862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:30.674865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:30.674985 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:30.675249 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:30.675251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:30.675253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:30.675563 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:30.675770 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:30.675979 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 13:14:30.676130 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":208,"threads_started":1}
I20260619 13:14:30.677067 8437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:30.792371 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 8307
I20260619 13:14:30.808885 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 7926
I20260619 13:14:30.809989 7998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:30.919934 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 7926
2026-06-19T13:14:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5634 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:14:31 dist-test-slave-mkt3 krb5kdc[8580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:14:31 dist-test-slave-mkt3 krb5kdc[8580](info): set up 2 sockets
Jun 19 13:14:31 dist-test-slave-mkt3 krb5kdc[8580](info): commencing operation
krb5kdc: starting...
W20260619 13:14:31.926507 388 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:14:32 dist-test-slave-mkt3 krb5kdc[8580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:14:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:14:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:14:32.123353 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35289
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:32.427107 8593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:32.427398 8593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:32.427515 8593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:32.434552 8593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:32.434660 8593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:32.434710 8593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:32.434785 8593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:32.434842 8593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:14:32.444440 8593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:32.445634 8593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:32.446918 8593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:32.457702 8601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:32.457728 8599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:32.458005 8598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:32.458464 8593 server_base.cc:1034] running on GCE node
I20260619 13:14:32.459151 8593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:32.460436 8593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:32.461632 8593 hybrid_clock.cc:648] HybridClock initialized: now 1781874872461590 us; error 27 us; skew 500 ppm
I20260619 13:14:32.461964 8593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T13:14:32Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1610 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:14:32 dist-test-slave-mkt3 krb5kdc[8607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:14:32 dist-test-slave-mkt3 krb5kdc[8607](info): set up 2 sockets
Jun 19 13:14:32 dist-test-slave-mkt3 krb5kdc[8607](info): commencing operation
krb5kdc: starting...
W20260619 13:14:33.538367 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:14:33 dist-test-slave-mkt3 krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:14:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:14:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:14:33.727077 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37475
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:34.031900 8620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:34.032202 8620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:34.032299 8620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:34.039467 8620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:34.039569 8620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:34.039613 8620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:34.039667 8620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:34.039708 8620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:14:34.049252 8620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:34.050530 8620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:34.051748 8620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:34.062862 8625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:34.062855 8626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:34.062883 8628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:34.063721 8620 server_base.cc:1034] running on GCE node
I20260619 13:14:34.064482 8620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:34.065827 8620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:34.067063 8620 hybrid_clock.cc:648] HybridClock initialized: now 1781874874067012 us; error 42 us; skew 500 ppm
I20260619 13:14:34.067394 8620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:34 dist-test-slave-mkt3 krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:34.070264 8620 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:14:34.071583 8620 webserver.cc:460] Webserver started at http://127.0.97.62:43001/ using document root <none> and password file <none>
I20260619 13:14:34.072212 8620 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:34.072314 8620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:34.072600 8620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:34.075449 8620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "ed47c801c9084a26994b3e74402aae4d"
format_stamp: "Formatted at 2026-06-19 13:14:34 on dist-test-slave-mkt3"
server_key: "0dd8ecee0ddcdb013d6f2fde6f7bb1b2"
server_key_iv: "40ea642b85eaa52a0d62f2a1658b0cb1"
server_key_version: "encryptionkey@0"
I20260619 13:14:34.076185 8620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "ed47c801c9084a26994b3e74402aae4d"
format_stamp: "Formatted at 2026-06-19 13:14:34 on dist-test-slave-mkt3"
server_key: "0dd8ecee0ddcdb013d6f2fde6f7bb1b2"
server_key_iv: "40ea642b85eaa52a0d62f2a1658b0cb1"
server_key_version: "encryptionkey@0"
I20260619 13:14:34.081065 8620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 13:14:34.084465 8635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:34.085781 8620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:14:34.085961 8620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "ed47c801c9084a26994b3e74402aae4d"
format_stamp: "Formatted at 2026-06-19 13:14:34 on dist-test-slave-mkt3"
server_key: "0dd8ecee0ddcdb013d6f2fde6f7bb1b2"
server_key_iv: "40ea642b85eaa52a0d62f2a1658b0cb1"
server_key_version: "encryptionkey@0"
I20260619 13:14:34.086134 8620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:34.097997 8620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:34.101473 8620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:34.101706 8620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:34.117410 8620 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37475
I20260619 13:14:34.117451 8686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37475 every 8 connection(s)
I20260619 13:14:34.118980 8620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:14:34.122460 8687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:34.124238 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 8620
I20260619 13:14:34.124444 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:14:34.125233 388 external_mini_cluster.cc:1428] Setting key 27f2c6c427f6f12b174505f445519b98
I20260619 13:14:34.134308 8687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d: Bootstrap starting.
I20260619 13:14:34.137858 8687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:34.138960 8687 log.cc:824] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:14:34 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:34.141569 8687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d: No bootstrap required, opened a new log
I20260619 13:14:34.146337 8690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:34.127173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.0.97.62:37475)
0619 13:14:34.127880 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:34.127891 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:34.127931 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:14:34.129514 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:34.129535 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:34.129933 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:34.130235 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:34.131855 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:34.133122 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:34.134844 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:34.135460 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:34.135584 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:34.139896 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:34.139903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:34.139920 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:34.139956 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:34.142726 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:34.143308 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:34.143314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:34.143319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:34.143473 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:34.143821 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:34.143825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:34.143827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:34.144318 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:34.144598 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:34.144973 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 13:14:34.145359 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":305,"threads_started":1}
I20260619 13:14:34.149040 8687 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47c801c9084a26994b3e74402aae4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37475 } }
I20260619 13:14:34.149374 8687 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:34.149468 8687 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c801c9084a26994b3e74402aae4d, State: Initialized, Role: FOLLOWER
I20260619 13:14:34.150036 8687 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47c801c9084a26994b3e74402aae4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37475 } }
I20260619 13:14:34.150243 8687 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:14:34.150377 8687 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:14:34.150520 8687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:34.152518 8687 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47c801c9084a26994b3e74402aae4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37475 } }
I20260619 13:14:34.152964 8687 leader_election.cc:302] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed47c801c9084a26994b3e74402aae4d; no voters:
I20260619 13:14:34.153483 8687 leader_election.cc:288] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:14:34.153589 8692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:34.154173 8692 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 1 LEADER]: Becoming Leader. State: Replica: ed47c801c9084a26994b3e74402aae4d, State: Running, Role: LEADER
I20260619 13:14:34.154698 8692 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47c801c9084a26994b3e74402aae4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37475 } }
I20260619 13:14:34.155529 8687 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:34.157287 8694 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed47c801c9084a26994b3e74402aae4d. Latest consensus state: current_term: 1 leader_uuid: "ed47c801c9084a26994b3e74402aae4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47c801c9084a26994b3e74402aae4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37475 } } }
I20260619 13:14:34.157270 8693 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed47c801c9084a26994b3e74402aae4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47c801c9084a26994b3e74402aae4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37475 } } }
I20260619 13:14:34.157660 8694 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:34.157660 8693 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:34.159029 8698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:14:34.163537 8698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:14:34.171387 8698 catalog_manager.cc:1347] Generated new cluster ID: 3c48c60828184feca24f1487d81ad861
I20260619 13:14:34.171489 8698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:14:34.196527 8698 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:14:34.197814 8698 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:14:34.211275 8698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d: Generated new TSK 0
I20260619 13:14:34.212347 8698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:14:34.269073 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37475
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:34.573994 8713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:34.574261 8713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:34.574393 8713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:34.581332 8713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:34.581430 8713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:34.581585 8713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:14:34.591259 8713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:34.592419 8713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:34.593645 8713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:34.605962 8718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:34.605906 8719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:34.605916 8721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:34.606720 8713 server_base.cc:1034] running on GCE node
I20260619 13:14:34.607266 8713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:34.608095 8713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:34.609304 8713 hybrid_clock.cc:648] HybridClock initialized: now 1781874874609255 us; error 41 us; skew 500 ppm
I20260619 13:14:34.609586 8713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:34 dist-test-slave-mkt3 krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:34.612440 8713 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:14:34.613704 8713 webserver.cc:460] Webserver started at http://127.0.97.1:40449/ using document root <none> and password file <none>
I20260619 13:14:34.614365 8713 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:34.614475 8713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:34.614746 8713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:34.617388 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "89b91e8868474f26ac301faed3649078"
format_stamp: "Formatted at 2026-06-19 13:14:34 on dist-test-slave-mkt3"
server_key: "0a17370da2754f3b3e45c410ff0c0dee"
server_key_iv: "57b5c55311ca8b6a3cc289d8afd212a6"
server_key_version: "encryptionkey@0"
I20260619 13:14:34.618117 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "89b91e8868474f26ac301faed3649078"
format_stamp: "Formatted at 2026-06-19 13:14:34 on dist-test-slave-mkt3"
server_key: "0a17370da2754f3b3e45c410ff0c0dee"
server_key_iv: "57b5c55311ca8b6a3cc289d8afd212a6"
server_key_version: "encryptionkey@0"
I20260619 13:14:34.622747 8713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 13:14:34.625823 8728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:34.627079 8713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 13:14:34.627218 8713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "89b91e8868474f26ac301faed3649078"
format_stamp: "Formatted at 2026-06-19 13:14:34 on dist-test-slave-mkt3"
server_key: "0a17370da2754f3b3e45c410ff0c0dee"
server_key_iv: "57b5c55311ca8b6a3cc289d8afd212a6"
server_key_version: "encryptionkey@0"
I20260619 13:14:34.627403 8713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:34.647429 8713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:34.650991 8713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:34.651224 8713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:34.652069 8713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:34.653244 8713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:34.653321 8713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:34.653429 8713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:34.653478 8713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:34.681676 8713 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43621
I20260619 13:14:34.681737 8840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43621 every 8 connection(s)
I20260619 13:14:34.682876 8713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:14:34.687665 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 8713
I20260619 13:14:34.687865 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:14:34.688591 388 external_mini_cluster.cc:1428] Setting key 203d1d27885f6511146fee3ad52627c4
Jun 19 13:14:34 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:34.702834 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:34.685108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35883 (local address 127.0.97.62:37475)
0619 13:14:34.685494 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:34.685498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:34.686384 (+ 886us) server_negotiation.cc:407] Connection header received
0619 13:14:34.688537 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:34.688542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:34.688609 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:34.688715 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:34.690563 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:34.691275 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:34.693345 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:34.693708 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:34.693792 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:34.697462 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:34.697466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:34.697469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:34.697500 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:34.700133 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:34.700777 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:34.700782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:34.700785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:34.700905 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:34.701255 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:34.701259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:34.701262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:34.701568 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:34.701698 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:34.702311 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 13:14:34.702500 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":202,"threads_started":1}
I20260619 13:14:34.703622 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:34.685542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:35883)
0619 13:14:34.686226 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:34.686277 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:34.688221 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:34.688979 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:34.688990 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:34.689628 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:34.690405 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:34.690426 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:34.691424 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:34.691429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:34.693220 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:34.693234 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:34.693916 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:34.693922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:34.694164 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:34.694803 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:34.694828 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:34.697243 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:34.700252 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:34.700259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:34.700277 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:34.700663 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:34.700986 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:34.700989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:34.700992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:34.701161 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:34.701662 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:34.701668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:34.701988 (+ 320us) client_negotiation.cc:769] Sending connection context
0619 13:14:34.702200 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 13:14:34.702556 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":268,"threads_started":1}
I20260619 13:14:34.704967 8841 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37475
I20260619 13:14:34.705228 8841 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:34.705907 8841 heartbeater.cc:511] Master 127.0.97.62:37475 requested a full tablet report, sending...
I20260619 13:14:34.707949 8652 ts_manager.cc:194] Registered new tserver with Master: 89b91e8868474f26ac301faed3649078 (127.0.97.1:43621)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:14:34.709385 8652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:14:34.732749 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37475
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:35.046090 8847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:35.046381 8847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:35.046522 8847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:35.053722 8847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:35.053865 8847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:35.054024 8847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:14:35.063932 8847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:35.065177 8847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:35.066457 8847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:35.078620 8852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:35.078676 8853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:35.078706 8855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:35.079831 8847 server_base.cc:1034] running on GCE node
I20260619 13:14:35.080441 8847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:35.081358 8847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:35.082543 8847 hybrid_clock.cc:648] HybridClock initialized: now 1781874875082496 us; error 44 us; skew 500 ppm
I20260619 13:14:35.082851 8847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:35 dist-test-slave-mkt3 krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:35.085831 8847 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:14:35.087188 8847 webserver.cc:460] Webserver started at http://127.0.97.2:45363/ using document root <none> and password file <none>
I20260619 13:14:35.087858 8847 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:35.087971 8847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:35.088255 8847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:35.091116 8847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "4066cc379a8e4ee384d44f9e25b0ef2d"
format_stamp: "Formatted at 2026-06-19 13:14:35 on dist-test-slave-mkt3"
server_key: "4f59cdc1ee9408857ad73ed383259e24"
server_key_iv: "02746d3ed6aaf623f23c855d9bc93163"
server_key_version: "encryptionkey@0"
I20260619 13:14:35.091857 8847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4066cc379a8e4ee384d44f9e25b0ef2d"
format_stamp: "Formatted at 2026-06-19 13:14:35 on dist-test-slave-mkt3"
server_key: "4f59cdc1ee9408857ad73ed383259e24"
server_key_iv: "02746d3ed6aaf623f23c855d9bc93163"
server_key_version: "encryptionkey@0"
I20260619 13:14:35.096652 8847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:14:35.099988 8862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:35.101279 8847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:14:35.101420 8847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "4066cc379a8e4ee384d44f9e25b0ef2d"
format_stamp: "Formatted at 2026-06-19 13:14:35 on dist-test-slave-mkt3"
server_key: "4f59cdc1ee9408857ad73ed383259e24"
server_key_iv: "02746d3ed6aaf623f23c855d9bc93163"
server_key_version: "encryptionkey@0"
I20260619 13:14:35.101599 8847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:35.122123 8847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:35.126160 8847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:35.126410 8847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:35.127352 8847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:35.128608 8847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:35.128684 8847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:35.128793 8847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:35.128841 8847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:35.157680 8847 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40367
I20260619 13:14:35.157727 8974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40367 every 8 connection(s)
I20260619 13:14:35.158947 8847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:14:35.161047 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 8847
I20260619 13:14:35.161183 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:14:35.161715 388 external_mini_cluster.cc:1428] Setting key 6573e7ebc4be22af50fd14f9a90fb40e
Jun 19 13:14:35 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:35.180186 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:35.161511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33315 (local address 127.0.97.62:37475)
0619 13:14:35.161706 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:35.161712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:35.163071 (+ 1359us) server_negotiation.cc:407] Connection header received
0619 13:14:35.164997 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:35.165003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:35.165095 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:35.165199 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:35.166709 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:35.167638 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:35.169221 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:35.169789 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:35.169938 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:35.173551 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:35.173556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:35.173561 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:35.173597 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:35.176664 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:35.177512 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:35.177518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:35.177521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:35.177707 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:35.178333 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:35.178339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:35.178342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:35.178663 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:35.178794 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:35.179738 (+ 944us) server_negotiation.cc:299] Negotiation successful
0619 13:14:35.179924 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:14:35.181777 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:35.162137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:33315)
0619 13:14:35.162896 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:35.162948 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:35.164716 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:35.165401 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:35.165408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:35.165983 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:35.166548 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:35.166570 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:35.167771 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:35.167775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:35.169086 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:35.169099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:35.170040 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:35.170047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:35.170334 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:35.171073 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:35.171096 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:35.173319 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:35.176838 (+ 3519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:35.176845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:35.176859 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:35.177366 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:35.177878 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:35.177883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:35.177887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:35.178185 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:35.178790 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:35.178799 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:35.179300 (+ 501us) client_negotiation.cc:769] Sending connection context
0619 13:14:35.179649 (+ 349us) client_negotiation.cc:240] Negotiation successful
0619 13:14:35.180259 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":260,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:14:35.183369 8975 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37475
I20260619 13:14:35.183701 8975 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:35.184410 8975 heartbeater.cc:511] Master 127.0.97.62:37475 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 13:14:35.185853 8652 ts_manager.cc:194] Registered new tserver with Master: 4066cc379a8e4ee384d44f9e25b0ef2d (127.0.97.2:40367)
I20260619 13:14:35.186884 8652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:14:35.208981 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37475
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:35.514158 8980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:35.514457 8980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:35.514585 8980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:35.521766 8980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:35.521934 8980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:35.522092 8980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:14:35.532395 8980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:35.533669 8980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:35.534998 8980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:35.547540 8986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:35.547559 8988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:35.547602 8985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:35.548429 8980 server_base.cc:1034] running on GCE node
I20260619 13:14:35.549052 8980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:35.550036 8980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:35.551261 8980 hybrid_clock.cc:648] HybridClock initialized: now 1781874875551190 us; error 48 us; skew 500 ppm
I20260619 13:14:35.551563 8980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:35 dist-test-slave-mkt3 krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:35.562448 8980 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:14:35.563930 8980 webserver.cc:460] Webserver started at http://127.0.97.3:33631/ using document root <none> and password file <none>
I20260619 13:14:35.564610 8980 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:35.564710 8980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:35.564975 8980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:35.567839 8980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "a7c0ad70da1d4985a73f24dd5fd79913"
format_stamp: "Formatted at 2026-06-19 13:14:35 on dist-test-slave-mkt3"
server_key: "af15d665133b10a541e042306dadf0b6"
server_key_iv: "7f9df6defcb55877b0ea0930778cdb54"
server_key_version: "encryptionkey@0"
I20260619 13:14:35.568554 8980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a7c0ad70da1d4985a73f24dd5fd79913"
format_stamp: "Formatted at 2026-06-19 13:14:35 on dist-test-slave-mkt3"
server_key: "af15d665133b10a541e042306dadf0b6"
server_key_iv: "7f9df6defcb55877b0ea0930778cdb54"
server_key_version: "encryptionkey@0"
I20260619 13:14:35.573451 8980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:14:35.576957 8995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:35.578254 8980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:14:35.578377 8980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "a7c0ad70da1d4985a73f24dd5fd79913"
format_stamp: "Formatted at 2026-06-19 13:14:35 on dist-test-slave-mkt3"
server_key: "af15d665133b10a541e042306dadf0b6"
server_key_iv: "7f9df6defcb55877b0ea0930778cdb54"
server_key_version: "encryptionkey@0"
I20260619 13:14:35.578538 8980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:35.601550 8980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:35.605396 8980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:35.605623 8980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:35.606599 8980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:35.607841 8980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:35.607911 8980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:35.608016 8980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:35.608063 8980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:35.636487 8980 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34071
I20260619 13:14:35.636534 9107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34071 every 8 connection(s)
I20260619 13:14:35.637669 8980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:14:35.647253 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 8980
I20260619 13:14:35.647403 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:14:35.647814 388 external_mini_cluster.cc:1428] Setting key 853ffc4f39113a8f6bca681a4787da9c
Jun 19 13:14:35 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:35.655563 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:35.640050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54277 (local address 127.0.97.62:37475)
0619 13:14:35.640248 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:35.640254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:35.641425 (+ 1171us) server_negotiation.cc:407] Connection header received
0619 13:14:35.643157 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:35.643162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:35.643228 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:35.643314 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:35.644666 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:35.645398 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:35.646923 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:35.647321 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:35.647447 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:35.650749 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:35.650754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:35.650757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:35.650788 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:35.652888 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:35.653624 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:35.653629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:35.653631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:35.653750 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:35.654123 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:35.654126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:35.654128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:35.654383 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:35.654486 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:35.655126 (+ 640us) server_negotiation.cc:299] Negotiation successful
0619 13:14:35.655334 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:14:35.656466 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:35.640582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:54277)
0619 13:14:35.641266 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:35.641313 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:35.642905 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:35.643495 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:35.643503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:35.644006 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:35.644539 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:35.644556 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:35.645516 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:35.645520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:35.646781 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:35.646793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:35.647584 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:35.647591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:35.647807 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:35.648405 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:35.648427 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:35.650576 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:35.653004 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:35.653011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:35.653024 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:35.653490 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:35.653831 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:35.653834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:35.653859 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:35.654034 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:35.654470 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:35.654475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:35.654783 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 13:14:35.654997 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 13:14:35.655417 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":246,"threads_started":1}
I20260619 13:14:35.657688 9108 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37475
I20260619 13:14:35.657994 9108 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:35.658644 9108 heartbeater.cc:511] Master 127.0.97.62:37475 requested a full tablet report, sending...
I20260619 13:14:35.659961 8652 ts_manager.cc:194] Registered new tserver with Master: a7c0ad70da1d4985a73f24dd5fd79913 (127.0.97.3:34071)
I20260619 13:14:35.660756 8652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54277
I20260619 13:14:35.662626 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 13:14:35.712221 8841 heartbeater.cc:503] Master 127.0.97.62:37475 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 13:14:35.782611 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 8713
Jun 19 13:14:35 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:14:35.797683 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:35.783614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50344 (local address 127.0.97.1:43621)
0619 13:14:35.784179 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:35.784187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:35.784216 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:14:35.784275 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:35.784282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:35.784617 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:35.784772 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:35.786127 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:35.787158 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:35.788349 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:35.788897 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:35.788995 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:35.792777 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:35.792788 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:35.792803 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:35.792840 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:35.795332 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:35.795962 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:35.795970 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:35.795972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:35.796098 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:35.796413 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:35.796417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:35.796420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:35.796766 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:35.796995 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:35.797197 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 13:14:35.797392 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":275,"threads_started":1}
I20260619 13:14:35.798956 8836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:35.879082 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 8713
I20260619 13:14:35.891769 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 8847
Jun 19 13:14:35 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:14:35.906248 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:35.892523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.0.97.2:40367)
0619 13:14:35.893036 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:35.893043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:35.893069 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:14:35.893127 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:35.893134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:35.893342 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:35.893473 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:35.894601 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:35.895642 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:35.896847 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:35.897392 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:35.897495 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:35.901182 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:35.901194 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:35.901209 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:35.901248 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:35.903891 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:35.904481 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:35.904488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:35.904490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:35.904609 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:35.904881 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:35.904884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:35.904885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:35.905243 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:35.905495 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:35.905723 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 13:14:35.905966 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":260,"threads_started":1}
I20260619 13:14:35.907464 8970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:35.987553 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 8847
I20260619 13:14:36.000417 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 8980
Jun 19 13:14:36 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:14:36.016325 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:36.001330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.0.97.3:34071)
0619 13:14:36.001958 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:36.001966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:36.001993 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:14:36.002050 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:36.002057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:36.002272 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:36.002417 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:36.003674 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:36.004801 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:36.006433 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:36.007101 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:36.007226 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:36.011244 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:36.011258 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:36.011272 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:36.011312 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:36.013783 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:36.014412 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:36.014420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:36.014422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:36.014546 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:36.014886 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:36.014890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:36.014894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:36.015216 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:36.015492 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:36.015773 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 13:14:36.016017 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":289,"threads_started":1}
I20260619 13:14:36.017486 9103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:36.097209 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 8980
I20260619 13:14:36.109464 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 8620
I20260619 13:14:36.110625 8682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:36.212402 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 8620
I20260619 13:14:36.228760 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37475
--webserver_interface=127.0.97.62
--webserver_port=43001
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:36.530936 9125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:36.531230 9125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:36.531359 9125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:36.538493 9125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:36.538589 9125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:36.538646 9125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:36.538723 9125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:36.538775 9125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:14:36.548228 9125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:36.549470 9125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:36.550742 9125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:36.561693 9133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:36.561757 9130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:36.561748 9131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:36.562623 9125 server_base.cc:1034] running on GCE node
I20260619 13:14:36.563388 9125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:36.564702 9125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:36.565924 9125 hybrid_clock.cc:648] HybridClock initialized: now 1781874876565873 us; error 33 us; skew 500 ppm
I20260619 13:14:36.566269 9125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:36 dist-test-slave-mkt3 krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:36.569113 9125 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:14:36.570524 9125 webserver.cc:460] Webserver started at http://127.0.97.62:43001/ using document root <none> and password file <none>
I20260619 13:14:36.571163 9125 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:36.571287 9125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:36.576398 9125 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 13:14:36.579031 9140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:36.580335 9125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:14:36.580488 9125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "ed47c801c9084a26994b3e74402aae4d"
format_stamp: "Formatted at 2026-06-19 13:14:34 on dist-test-slave-mkt3"
server_key: "0dd8ecee0ddcdb013d6f2fde6f7bb1b2"
server_key_iv: "40ea642b85eaa52a0d62f2a1658b0cb1"
server_key_version: "encryptionkey@0"
I20260619 13:14:36.581461 9125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:36.592994 9125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:36.596954 9125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:36.597177 9125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:36.613477 9125 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37475
I20260619 13:14:36.613497 9191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37475 every 8 connection(s)
I20260619 13:14:36.614928 9125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:14:36.616360 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 9125
I20260619 13:14:36.617271 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43621
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37475
--webserver_port=40449
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:14:36.623359 9192 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:14:36.626932 9192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d: Bootstrap starting.
I20260619 13:14:36.645090 9192 log.cc:824] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:36.653064 9192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:14:36.653529 9192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d: Bootstrap complete.
I20260619 13:14:36.661900 9192 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47c801c9084a26994b3e74402aae4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37475 } }
I20260619 13:14:36.662695 9192 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed47c801c9084a26994b3e74402aae4d, State: Initialized, Role: FOLLOWER
I20260619 13:14:36.663265 9192 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47c801c9084a26994b3e74402aae4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37475 } }
I20260619 13:14:36.663460 9192 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:14:36.663566 9192 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:14:36.663713 9192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:14:36.667127 9192 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47c801c9084a26994b3e74402aae4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37475 } }
I20260619 13:14:36.667555 9192 leader_election.cc:302] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed47c801c9084a26994b3e74402aae4d; no voters:
I20260619 13:14:36.668110 9192 leader_election.cc:288] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 13:14:36.668269 9196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 2 FOLLOWER]: Leader election won for term 2
I20260619 13:14:36.669291 9196 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [term 2 LEADER]: Becoming Leader. State: Replica: ed47c801c9084a26994b3e74402aae4d, State: Running, Role: LEADER
I20260619 13:14:36.669822 9196 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47c801c9084a26994b3e74402aae4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37475 } }
I20260619 13:14:36.670182 9192 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:36.672580 9197 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ed47c801c9084a26994b3e74402aae4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47c801c9084a26994b3e74402aae4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37475 } } }
I20260619 13:14:36.672916 9197 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:36.673328 9198 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed47c801c9084a26994b3e74402aae4d. Latest consensus state: current_term: 2 leader_uuid: "ed47c801c9084a26994b3e74402aae4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed47c801c9084a26994b3e74402aae4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37475 } } }
I20260619 13:14:36.673588 9198 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:36.675678 9202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:14:36.681038 9202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:14:36.682390 9202 catalog_manager.cc:1259] Loaded cluster ID: 3c48c60828184feca24f1487d81ad861
I20260619 13:14:36.682526 9202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:14:36.686462 9202 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:14:36.687830 9202 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ed47c801c9084a26994b3e74402aae4d: Loaded TSK: 0
I20260619 13:14:36.689307 9202 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 13:14:36.936303 9194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:36.936585 9194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:36.936719 9194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:36.943794 9194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:36.943913 9194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:36.944054 9194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:14:36.956269 9194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40449
--tserver_master_addrs=127.0.97.62:37475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:36.957633 9194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:36.958966 9194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:36.973315 9219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:36.973387 9221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:36.973462 9218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:36.974828 9194 server_base.cc:1034] running on GCE node
I20260619 13:14:36.975433 9194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:36.976370 9194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:36.977576 9194 hybrid_clock.cc:648] HybridClock initialized: now 1781874876977509 us; error 61 us; skew 500 ppm
I20260619 13:14:36.977910 9194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:36 dist-test-slave-mkt3 krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874876, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:36.980922 9194 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 13:14:36.982373 9194 webserver.cc:460] Webserver started at http://127.0.97.1:40449/ using document root <none> and password file <none>
I20260619 13:14:36.983031 9194 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:36.983160 9194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:36.988377 9194 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 13:14:36.990935 9228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:36.992245 9194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:14:36.992377 9194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "89b91e8868474f26ac301faed3649078"
format_stamp: "Formatted at 2026-06-19 13:14:34 on dist-test-slave-mkt3"
server_key: "0a17370da2754f3b3e45c410ff0c0dee"
server_key_iv: "57b5c55311ca8b6a3cc289d8afd212a6"
server_key_version: "encryptionkey@0"
I20260619 13:14:36.993249 9194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:37.026786 9194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:37.031080 9194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:37.031332 9194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:37.032286 9194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:37.033542 9194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:37.033630 9194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:37.033738 9194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:37.033810 9194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:37.062474 9194 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43621
I20260619 13:14:37.062537 9340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43621 every 8 connection(s)
I20260619 13:14:37.063832 9194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:14:37.069254 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 9194
I20260619 13:14:37.069803 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40367
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37475
--webserver_port=45363
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:37.080802 9344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:37.066622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:36331)
0619 13:14:37.067403 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:37.067439 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:37.069031 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:37.070480 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:37.070491 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:37.071077 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:37.071600 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.071616 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.073287 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.073291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.074533 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.074546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.075516 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.075523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.075770 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.076398 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:37.076420 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:37.079853 (+ 3433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":317,"threads_started":1}
W20260619 13:14:37.081337 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:37.066483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36331 (local address 127.0.97.62:37475)
0619 13:14:37.067308 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:37.067323 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:37.067549 (+ 226us) server_negotiation.cc:407] Connection header received
0619 13:14:37.069562 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:37.069575 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:37.069970 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:37.070311 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:37.071808 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.073128 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.074662 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.075309 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.075472 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.080249 (+ 4777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36331: BlockingRecv error: recv got EOF from 127.0.97.1:36331 (error 108)
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":323,"threads_started":1}
W20260619 13:14:37.081578 9341 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37475 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:37475: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:37.089586 9344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:37.082158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:43725)
0619 13:14:37.082318 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:37.082336 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:37.082433 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:37.082715 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:37.082718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:37.082978 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:37.083104 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.083111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.084350 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.084354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.085243 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.085256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.086170 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.086174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.086332 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.086831 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:37.086847 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:37.089402 (+ 2555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 13:14:37.090011 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:37.082305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43725 (local address 127.0.97.62:37475)
0619 13:14:37.082466 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:37.082472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:37.082488 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:14:37.082536 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:37.082541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:37.082602 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:37.082687 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:37.083219 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.084232 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.085359 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.086034 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.086156 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.089627 (+ 3471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43725: BlockingRecv error: recv got EOF from 127.0.97.1:43725 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:37.098151 9344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:37.090268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:48943)
0619 13:14:37.090435 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:37.090454 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:37.090557 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:37.090825 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:37.090830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:37.091178 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:37.091344 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.091354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.092216 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.092221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.093677 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.093690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.094368 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.094374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.094547 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.095058 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:37.095079 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:37.097877 (+ 2798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 13:14:37.098435 9341 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:14:37.098626 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:37.090323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48943 (local address 127.0.97.62:37475)
0619 13:14:37.090464 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:37.090468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:37.090501 (+ 33us) server_negotiation.cc:407] Connection header received
0619 13:14:37.090652 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:37.090655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:37.090715 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:37.090788 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:37.091450 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.092099 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.093794 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.094245 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.094332 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.098089 (+ 3757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48943: BlockingRecv error: recv got EOF from 127.0.97.1:48943 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 13:14:37.381918 9345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:37.382198 9345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:37.382349 9345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:37.389448 9345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:37.389559 9345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:37.389707 9345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:14:37.399251 9345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45363
--tserver_master_addrs=127.0.97.62:37475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:37.400481 9345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:37.401712 9345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:37.413908 9350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:37.413888 9351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:37.413820 9353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:37.414544 9345 server_base.cc:1034] running on GCE node
I20260619 13:14:37.415149 9345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:37.416023 9345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:37.417213 9345 hybrid_clock.cc:648] HybridClock initialized: now 1781874877417162 us; error 37 us; skew 500 ppm
I20260619 13:14:37.417506 9345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874877, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:37.420419 9345 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 13:14:37.421705 9345 webserver.cc:460] Webserver started at http://127.0.97.2:45363/ using document root <none> and password file <none>
I20260619 13:14:37.422406 9345 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:37.422533 9345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:37.427829 9345 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 13:14:37.430562 9360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:37.431903 9345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:14:37.432052 9345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "4066cc379a8e4ee384d44f9e25b0ef2d"
format_stamp: "Formatted at 2026-06-19 13:14:35 on dist-test-slave-mkt3"
server_key: "4f59cdc1ee9408857ad73ed383259e24"
server_key_iv: "02746d3ed6aaf623f23c855d9bc93163"
server_key_version: "encryptionkey@0"
I20260619 13:14:37.432993 9345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:37.459348 9345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:37.463805 9345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:37.464064 9345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:37.465015 9345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:37.466305 9345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:37.466399 9345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:37.466497 9345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:37.466567 9345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:37.495187 9345 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40367
I20260619 13:14:37.495251 9472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40367 every 8 connection(s)
I20260619 13:14:37.496655 9345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:14:37.501245 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 9345
I20260619 13:14:37.501809 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34071
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37475
--webserver_port=33631
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 13:14:37.511783 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:37.499163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57757 (local address 127.0.97.62:37475)
0619 13:14:37.499326 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:37.499330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:37.500459 (+ 1129us) server_negotiation.cc:407] Connection header received
0619 13:14:37.502392 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:37.502398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:37.502478 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:37.502561 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:37.504142 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.504840 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.506454 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.506864 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.506976 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.511543 (+ 4567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57757: BlockingRecv error: recv got EOF from 127.0.97.2:57757 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:14:37.512498 9475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:37.499627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:57757)
0619 13:14:37.500299 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:37.500337 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:37.502128 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:37.502813 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:37.502822 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:37.503366 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:37.503957 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.503976 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.505000 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.505004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.506307 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.506319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.507076 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.507084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.507315 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.507937 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:37.507959 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:37.511362 (+ 3403us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":259,"threads_started":1}
W20260619 13:14:37.513144 9473 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37475 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:37475: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:37.521409 9475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:37.513694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:45091)
0619 13:14:37.513878 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:37.513895 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:37.513992 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:37.514312 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:37.514315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:37.514641 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:37.514770 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.514778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.516099 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.516103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.516962 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.516982 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.517916 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.517921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.518050 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.518568 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:37.518584 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:37.521201 (+ 2617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 13:14:37.521667 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:37.513880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45091 (local address 127.0.97.62:37475)
0619 13:14:37.514062 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:37.514068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:37.514086 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:14:37.514135 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:37.514140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:37.514205 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:37.514299 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:37.514902 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.515933 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.517108 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.517722 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.517877 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.521434 (+ 3557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45091: BlockingRecv error: recv got EOF from 127.0.97.2:45091 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:37.529074 9475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:37.522097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:33703)
0619 13:14:37.522234 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:37.522251 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:37.522340 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:37.522631 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:37.522635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:37.522917 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:37.523047 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.523055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.523993 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.523998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.524893 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.524903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.525548 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.525553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.525696 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.526236 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:37.526252 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:37.528802 (+ 2550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 13:14:37.529343 9473 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:14:37.529531 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:37.522241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33703 (local address 127.0.97.62:37475)
0619 13:14:37.522408 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:37.522413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:37.522429 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:14:37.522472 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:37.522476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:37.522524 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:37.522597 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:37.523172 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.523870 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.525018 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.525423 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.525526 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.528989 (+ 3463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33703: BlockingRecv error: recv got EOF from 127.0.97.2:33703 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 13:14:37.816743 9476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:37.817047 9476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:37.817189 9476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:37.824329 9476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:37.824455 9476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:37.824615 9476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:14:37.834291 9476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33631
--tserver_master_addrs=127.0.97.62:37475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:37.835604 9476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:37.836856 9476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:37.849052 9481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:37.849088 9482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:37.849014 9484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:37.849822 9476 server_base.cc:1034] running on GCE node
I20260619 13:14:37.850438 9476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:37.851359 9476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:37.852533 9476 hybrid_clock.cc:648] HybridClock initialized: now 1781874877852478 us; error 55 us; skew 500 ppm
I20260619 13:14:37.852813 9476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874877, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:37.855922 9476 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 13:14:37.857336 9476 webserver.cc:460] Webserver started at http://127.0.97.3:33631/ using document root <none> and password file <none>
I20260619 13:14:37.858057 9476 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:37.858186 9476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:37.863417 9476 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:14:37.866125 9491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:37.867499 9476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:14:37.867656 9476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "a7c0ad70da1d4985a73f24dd5fd79913"
format_stamp: "Formatted at 2026-06-19 13:14:35 on dist-test-slave-mkt3"
server_key: "af15d665133b10a541e042306dadf0b6"
server_key_iv: "7f9df6defcb55877b0ea0930778cdb54"
server_key_version: "encryptionkey@0"
I20260619 13:14:37.868615 9476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:37.885046 9476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:37.889124 9476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:37.889385 9476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:37.890388 9476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:37.891649 9476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:37.891752 9476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:37.891860 9476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:37.891934 9476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:37.921206 9476 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34071
I20260619 13:14:37.921277 9603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34071 every 8 connection(s)
I20260619 13:14:37.922680 9476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:14:37.931156 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 9476
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 13:14:37.938505 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:37.925099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51215 (local address 127.0.97.62:37475)
0619 13:14:37.925250 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:37.925254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:37.926460 (+ 1206us) server_negotiation.cc:407] Connection header received
0619 13:14:37.928302 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:37.928307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:37.928387 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:37.928474 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:37.930064 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.930766 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.932584 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.933190 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.933318 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.938250 (+ 4932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51215: BlockingRecv error: recv got EOF from 127.0.97.3:51215 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 13:14:37.939215 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:37.925566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:51215)
0619 13:14:37.926283 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:37.926340 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:37.928014 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:37.928729 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:37.928738 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:37.929285 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:37.929897 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.929918 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.930917 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.930920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.932406 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.932422 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.933379 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.933384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.933597 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.934254 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:37.934277 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:37.938074 (+ 3797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":300,"threads_started":1}
W20260619 13:14:37.939924 9604 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37475 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:37475: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260619 13:14:37.946429 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:37.932083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.0.97.62:37475)
0619 13:14:37.932686 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:37.932690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:37.932705 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:14:37.932747 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:37.932750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:37.932801 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:37.932895 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:37.934515 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.935570 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.936938 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.937353 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.937459 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.940001 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:37.940012 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:37.940032 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:37.940076 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:37.943021 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:37.943795 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:37.943804 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:37.943814 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:37.944081 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:37.944535 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:37.944541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:37.944544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:37.945132 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:37.945453 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:37.945709 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 13:14:37.946080 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":363,"threads_started":1}
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:37.948414 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:37.940493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:56655)
0619 13:14:37.940648 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:37.940666 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:37.940758 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:37.941013 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:37.941018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:37.941350 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:37.941476 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.941484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.942518 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.942523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.943564 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.943579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.944460 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.944465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.944632 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.945102 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:37.945124 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:37.948218 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 13:14:37.948668 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:37.940559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56655 (local address 127.0.97.62:37475)
0619 13:14:37.940720 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:37.940725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:37.940739 (+ 14us) server_negotiation.cc:407] Connection header received
0619 13:14:37.940847 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:37.940852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:37.940909 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:37.940985 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:37.941597 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.942393 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.943732 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.944339 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.944454 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.948436 (+ 3982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56655: BlockingRecv error: recv got EOF from 127.0.97.3:56655 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:37 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:37.956010 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:37.949417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:39433)
0619 13:14:37.949564 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:37.949580 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:37.949673 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:37.950006 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:37.950010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:37.950285 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:37.950414 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.950422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.951340 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.951345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.952215 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:37.952225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.952841 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.952846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:37.953008 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.953434 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:37.953449 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:37.955749 (+ 2300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 13:14:37.956190 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:37.949500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39433 (local address 127.0.97.62:37475)
0619 13:14:37.949697 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:37.949702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:37.949718 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:14:37.949760 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:37.949784 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:37.949886 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:37.949968 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:37.950535 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.951213 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.952344 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:37.952718 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:37.952827 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:37.955967 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39433: BlockingRecv error: recv got EOF from 127.0.97.3:39433 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260619 13:14:37.956322 9604 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 13:14:38 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:38 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:38.107214 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:38.098952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:34223)
0619 13:14:38.099430 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:38.099446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:38.099607 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:38.099928 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:38.099933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:38.100393 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:38.100574 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:38.100583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:38.101594 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:38.101600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:38.102562 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:38.102575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:38.103303 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:38.103309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:38.103463 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:38.104076 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:38.104093 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:38.106917 (+ 2824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":254,"threads_started":1}
W20260619 13:14:38.107371 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:38.099049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34223 (local address 127.0.97.62:37475)
0619 13:14:38.099261 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:38.099291 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:38.099531 (+ 240us) server_negotiation.cc:407] Connection header received
0619 13:14:38.099727 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:38.099733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:38.099799 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:38.099883 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:38.100723 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:38.101457 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:38.102705 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:38.103135 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:38.103243 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:38.107143 (+ 3900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34223: BlockingRecv error: recv got EOF from 127.0.97.1:34223 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 13:14:38 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:38 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:38.538295 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:38.529905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:45021)
0619 13:14:38.530404 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:38.530421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:38.530538 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:38.530806 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:38.530809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:38.531251 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:38.531437 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:38.531452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:38.532456 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:38.532461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:38.533427 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:38.533438 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:38.534247 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:38.534251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:38.534378 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:38.534920 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:38.534935 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:38.538048 (+ 3113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":275,"threads_started":1}
W20260619 13:14:38.538429 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:38.529985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45021 (local address 127.0.97.62:37475)
0619 13:14:38.530172 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:38.530177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:38.530525 (+ 348us) server_negotiation.cc:407] Connection header received
0619 13:14:38.530632 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:38.530635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:38.530695 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:38.530778 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:38.531594 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:38.532317 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:38.533575 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:38.534055 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:38.534161 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:38.538232 (+ 4071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45021: BlockingRecv error: recv got EOF from 127.0.97.2:45021 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 13:14:38 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:38 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:38.965355 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:38.956931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:51153)
0619 13:14:38.957416 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:38.957435 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:38.957577 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:38.958092 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:38.958096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:38.958664 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:38.958906 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:38.958920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:38.960215 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:38.960219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:38.961398 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:38.961411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:38.962106 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:38.962110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:38.962222 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:38.962768 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:38.962785 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:38.965133 (+ 2348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":242,"threads_started":1}
W20260619 13:14:38.965484 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:38.957007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51153 (local address 127.0.97.62:37475)
0619 13:14:38.957178 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:38.957184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:38.957532 (+ 348us) server_negotiation.cc:407] Connection header received
0619 13:14:38.957696 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:38.957701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:38.957783 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:38.957921 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:38.959079 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:38.960061 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:38.961516 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:38.961975 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:38.962069 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:38.965309 (+ 3240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51153: BlockingRecv error: recv got EOF from 127.0.97.3:51153 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 13:14:39 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:39 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:39.115906 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:39.108138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:44107)
0619 13:14:39.108628 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:39.108644 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:39.108787 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:39.109085 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:39.109090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:39.109484 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:39.109621 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:39.109629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:39.110661 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:39.110666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:39.111552 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:39.111564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:39.112221 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:39.112226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:39.112374 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:39.112932 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:39.112954 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:39.115666 (+ 2712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":273,"threads_started":1}
W20260619 13:14:39.116060 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:39.108260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44107 (local address 127.0.97.62:37475)
0619 13:14:39.108480 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:39.108486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:39.108726 (+ 240us) server_negotiation.cc:407] Connection header received
0619 13:14:39.108899 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:39.108905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:39.108964 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:39.109040 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:39.109759 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:39.110528 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:39.111678 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:39.112078 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:39.112189 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:39.115847 (+ 3658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44107: BlockingRecv error: recv got EOF from 127.0.97.1:44107 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 13:14:39 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:39 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:39.546559 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:39.539061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:37561)
0619 13:14:39.539586 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:39.539602 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:39.539709 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:39.539976 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:39.539979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:39.540365 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:39.540500 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:39.540508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:39.541488 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:39.541491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:39.542436 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:39.542449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:39.543032 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:39.543036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:39.543136 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:39.543663 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:39.543682 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:39.546321 (+ 2639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":300,"threads_started":1}
W20260619 13:14:39.546672 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:39.539148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37561 (local address 127.0.97.62:37475)
0619 13:14:39.539320 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:39.539325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:39.539671 (+ 346us) server_negotiation.cc:407] Connection header received
0619 13:14:39.539806 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:39.539809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:39.539870 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:39.539947 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:39.540629 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:39.541377 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:39.542539 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:39.542925 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:39.543010 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:39.546498 (+ 3488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37561: BlockingRecv error: recv got EOF from 127.0.97.2:37561 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 13:14:39 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:39 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:39.973637 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:39.966155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:45313)
0619 13:14:39.966571 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:39.966587 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:39.966702 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:39.966974 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:39.966976 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:39.967340 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:39.967498 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:39.967505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:39.968431 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:39.968434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:39.969331 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:39.969343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:39.969998 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:39.970002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:39.970129 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:39.970623 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:39.970645 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:39.973296 (+ 2651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":221,"threads_started":1}
W20260619 13:14:39.973760 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:39.966305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45313 (local address 127.0.97.62:37475)
0619 13:14:39.966521 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:39.966526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:39.966651 (+ 125us) server_negotiation.cc:407] Connection header received
0619 13:14:39.966816 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:39.966819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:39.966877 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:39.966944 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:39.967601 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:39.968323 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:39.969449 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:39.969882 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:39.969987 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:39.973499 (+ 3512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45313: BlockingRecv error: recv got EOF from 127.0.97.3:45313 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 19 13:14:40 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:40 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:40.124620 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:40.116813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:46291)
0619 13:14:40.117297 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:40.117313 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:40.117439 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:40.117740 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:40.117745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:40.118107 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:40.118285 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:40.118295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:40.119273 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:40.119280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:40.120150 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:40.120161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:40.120857 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:40.120863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:40.121044 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:40.121586 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:40.121602 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:40.124247 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":243,"threads_started":1}
W20260619 13:14:40.124704 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:40.116880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46291 (local address 127.0.97.62:37475)
0619 13:14:40.117045 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:40.117051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:40.117393 (+ 342us) server_negotiation.cc:407] Connection header received
0619 13:14:40.117547 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:40.117552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:40.117618 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:40.117696 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:40.118423 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:40.119134 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:40.120282 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:40.120711 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:40.120822 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:40.124483 (+ 3661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46291: BlockingRecv error: recv got EOF from 127.0.97.1:46291 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 13:14:40 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:40 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:40.555310 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:40.547542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:35887)
0619 13:14:40.548034 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:40.548050 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:40.548184 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:40.548478 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:40.548483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:40.548827 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:40.548989 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:40.548998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:40.550025 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:40.550029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:40.550879 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:40.550891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:40.551548 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:40.551554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:40.551716 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:40.552297 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:40.552318 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:40.554959 (+ 2641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":270,"threads_started":1}
W20260619 13:14:40.555356 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:40.547595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35887 (local address 127.0.97.62:37475)
0619 13:14:40.547825 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:40.547831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:40.548141 (+ 310us) server_negotiation.cc:407] Connection header received
0619 13:14:40.548293 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:40.548297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:40.548355 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:40.548434 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:40.549133 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:40.549905 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:40.551009 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:40.551421 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:40.551526 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:40.555150 (+ 3624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35887: BlockingRecv error: recv got EOF from 127.0.97.2:35887 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 13:14:40 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:40 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:40.986078 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:40.974607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:58303)
0619 13:14:40.975086 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:40.975102 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:40.975225 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:40.975572 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:40.975575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:40.975890 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:40.976057 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:40.976065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:40.977101 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:40.977105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:40.978095 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:40.978108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:40.978702 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:40.978705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:40.978823 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:40.983181 (+ 4358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:40.983200 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:40.985797 (+ 2597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":236,"threads_started":1}
W20260619 13:14:40.986271 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:40.974697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58303 (local address 127.0.97.62:37475)
0619 13:14:40.974873 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:40.974878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:40.975167 (+ 289us) server_negotiation.cc:407] Connection header received
0619 13:14:40.975365 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:40.975369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:40.975450 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:40.975541 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:40.976192 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:40.976972 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:40.978207 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:40.978589 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:40.978678 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:40.986078 (+ 7400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58303: BlockingRecv error: recv got EOF from 127.0.97.3:58303 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 13:14:41 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:41 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:41.133044 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:41.125475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:40947)
0619 13:14:41.125962 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:41.125978 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:41.126101 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:41.126405 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:41.126410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:41.126772 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:41.126915 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:41.126924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:41.127877 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:41.127882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:41.128823 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:41.128835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:41.129502 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:41.129507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:41.129679 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:41.130259 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:41.130309 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:41.132742 (+ 2433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":249,"threads_started":1}
W20260619 13:14:41.133200 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:41.125558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40947 (local address 127.0.97.62:37475)
0619 13:14:41.125791 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:41.125798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:41.126053 (+ 255us) server_negotiation.cc:407] Connection header received
0619 13:14:41.126210 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:41.126215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:41.126280 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:41.126362 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:41.127054 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:41.127745 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:41.128973 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:41.129372 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:41.129478 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:41.132973 (+ 3495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40947: BlockingRecv error: recv got EOF from 127.0.97.1:40947 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 19 13:14:41 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:41 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:41.563882 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:41.556119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:42791)
0619 13:14:41.556571 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:41.556587 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:41.556706 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:41.557004 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:41.557009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:41.557382 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:41.557538 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:41.557546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:41.558541 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:41.558546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:41.559495 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:41.559508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:41.560241 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:41.560246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:41.560422 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:41.561005 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:41.561022 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:41.563584 (+ 2562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":256,"threads_started":1}
W20260619 13:14:41.563982 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:41.556204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42791 (local address 127.0.97.62:37475)
0619 13:14:41.556403 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:41.556410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:41.556657 (+ 247us) server_negotiation.cc:407] Connection header received
0619 13:14:41.556822 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:41.556827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:41.556885 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:41.556962 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:41.557667 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:41.558419 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:41.559622 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:41.560110 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:41.560216 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:41.563771 (+ 3555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42791: BlockingRecv error: recv got EOF from 127.0.97.2:42791 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 13:14:41 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:41 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:41.994498 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:41.986974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:56215)
0619 13:14:41.987425 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:41.987440 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:41.987551 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:41.987814 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:41.987817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:41.988151 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:41.988315 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:41.988323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:41.989288 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:41.989292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:41.990270 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:41.990283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:41.990892 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:41.990896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:41.991025 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:41.991550 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:41.991566 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:41.994195 (+ 2629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":229,"threads_started":1}
W20260619 13:14:41.994659 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:41.987026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56215 (local address 127.0.97.62:37475)
0619 13:14:41.987167 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:41.987171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:41.987505 (+ 334us) server_negotiation.cc:407] Connection header received
0619 13:14:41.987642 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:41.987646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:41.987706 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:41.987790 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:41.988425 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:41.989154 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:41.990388 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:41.990787 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:41.990877 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:41.994435 (+ 3558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56215: BlockingRecv error: recv got EOF from 127.0.97.3:56215 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 13:14:42 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:42 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:42.141499 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:42.133968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:37025)
0619 13:14:42.134451 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:42.134467 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:42.134576 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:42.134833 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:42.134836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:42.135188 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:42.135373 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:42.135382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:42.136314 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:42.136318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:42.137272 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:42.137284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:42.137982 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:42.137987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:42.138147 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:42.138675 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:42.138692 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:42.141208 (+ 2516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":225,"threads_started":1}
W20260619 13:14:42.141580 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:42.134036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37025 (local address 127.0.97.62:37475)
0619 13:14:42.134204 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:42.134209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:42.134549 (+ 340us) server_negotiation.cc:407] Connection header received
0619 13:14:42.134665 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:42.134668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:42.134727 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:42.134808 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:42.135492 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:42.136180 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:42.137382 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:42.137824 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:42.137952 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:42.141412 (+ 3460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37025: BlockingRecv error: recv got EOF from 127.0.97.1:37025 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 13:14:42 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:42 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:42.572755 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:42.564693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:51453)
0619 13:14:42.565166 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:42.565182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:42.565300 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:42.565600 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:42.565605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:42.566033 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:42.566249 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:42.566262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:42.567489 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:42.567493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:42.568486 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:42.568499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:42.569199 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:42.569203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:42.569328 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:42.569859 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:42.569886 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:42.572516 (+ 2630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":234,"threads_started":1}
W20260619 13:14:42.572892 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:42.564878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51453 (local address 127.0.97.62:37475)
0619 13:14:42.565076 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:42.565082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:42.565249 (+ 167us) server_negotiation.cc:407] Connection header received
0619 13:14:42.565415 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:42.565420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:42.565482 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:42.565557 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:42.566389 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:42.567358 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:42.568611 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:42.569068 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:42.569161 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:42.572682 (+ 3521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51453: BlockingRecv error: recv got EOF from 127.0.97.2:51453 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 13:14:42 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:42 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:43.003541 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:42.995508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:49541)
0619 13:14:42.995977 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:42.995991 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:42.996143 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:42.996445 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:42.996448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:42.996849 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:42.997021 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:42.997029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:42.998147 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:42.998151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:42.999046 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:42.999057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:42.999832 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:42.999838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:42.999997 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:43.000567 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:43.000585 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:43.003223 (+ 2638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":239,"threads_started":1}
W20260619 13:14:43.003685 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:42.995577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49541 (local address 127.0.97.62:37475)
0619 13:14:42.995739 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:42.995745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:42.996088 (+ 343us) server_negotiation.cc:407] Connection header received
0619 13:14:42.996268 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:42.996272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:42.996333 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:42.996407 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:42.997180 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:42.997991 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:42.999186 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:42.999703 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:42.999808 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:43.003427 (+ 3619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49541: BlockingRecv error: recv got EOF from 127.0.97.3:49541 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 13:14:43 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:43 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:43.149894 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:43.142329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:42501)
0619 13:14:43.142820 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:43.142835 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:43.142959 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:43.143264 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:43.143269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:43.143638 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:43.143777 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:43.143786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:43.144750 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:43.144754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:43.145622 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:43.145634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:43.146360 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:43.146366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:43.146518 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:43.147063 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:43.147080 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:43.149649 (+ 2569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":289,"threads_started":1}
W20260619 13:14:43.150077 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:43.142418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42501 (local address 127.0.97.62:37475)
0619 13:14:43.142585 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:43.142591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:43.142916 (+ 325us) server_negotiation.cc:407] Connection header received
0619 13:14:43.143063 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:43.143068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:43.143125 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:43.143200 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:43.143909 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:43.144621 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:43.145756 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:43.146217 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:43.146323 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:43.149829 (+ 3506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42501: BlockingRecv error: recv got EOF from 127.0.97.1:42501 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 13:14:43 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:43 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:43.581104 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:43.573550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:50581)
0619 13:14:43.574005 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:43.574021 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:43.574153 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:43.574423 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:43.574426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:43.574767 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:43.574912 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:43.574919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:43.575831 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:43.575835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:43.576762 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:43.576777 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:43.577435 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:43.577442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:43.577564 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:43.578162 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:43.578178 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:43.580870 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":236,"threads_started":1}
W20260619 13:14:43.581301 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:43.573658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50581 (local address 127.0.97.62:37475)
0619 13:14:43.573876 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:43.573882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:43.574109 (+ 227us) server_negotiation.cc:407] Connection header received
0619 13:14:43.574235 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:43.574252 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:43.574325 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:43.574398 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:43.575010 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:43.575698 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:43.576895 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:43.577292 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:43.577379 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:43.581092 (+ 3713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50581: BlockingRecv error: recv got EOF from 127.0.97.2:50581 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 13:14:44 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:44 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:44.012470 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:44.004553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:43697)
0619 13:14:44.005052 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:44.005069 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:44.005186 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:44.005470 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:44.005473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:44.005875 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:44.006023 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:44.006057 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:44.007059 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:44.007063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:44.007974 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:44.007986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:44.008612 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:44.008616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:44.008732 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:44.009262 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:44.009280 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:44.012196 (+ 2916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":267,"threads_started":1}
W20260619 13:14:44.012629 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:44.004625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43697 (local address 127.0.97.62:37475)
0619 13:14:44.004785 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:44.004790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:44.005135 (+ 345us) server_negotiation.cc:407] Connection header received
0619 13:14:44.005275 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:44.005278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:44.005336 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:44.005416 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:44.006176 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:44.006910 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:44.008085 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:44.008500 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:44.008593 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:44.012391 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43697: BlockingRecv error: recv got EOF from 127.0.97.3:43697 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 13:14:44 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:44 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:44.158407 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:44.150722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:43109)
0619 13:14:44.151223 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:44.151239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:44.151370 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:44.151659 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:44.151663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:44.152081 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:44.152238 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:44.152248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:44.153213 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:44.153217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:44.154232 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:44.154245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:44.154876 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:44.154881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:44.155031 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:44.155523 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:44.155539 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:44.158084 (+ 2545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":204,"threads_started":1}
W20260619 13:14:44.158492 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:44.150816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43109 (local address 127.0.97.62:37475)
0619 13:14:44.150998 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:44.151005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:44.151327 (+ 322us) server_negotiation.cc:407] Connection header received
0619 13:14:44.151478 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:44.151483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:44.151544 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:44.151620 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:44.152375 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:44.153079 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:44.154351 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:44.154742 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:44.154847 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:44.158292 (+ 3445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43109: BlockingRecv error: recv got EOF from 127.0.97.1:43109 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 13:14:44 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:44 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:44.589746 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:44.581953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:47159)
0619 13:14:44.582431 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:44.582447 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:44.582558 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:44.582838 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:44.582840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:44.583192 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:44.583369 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:44.583378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:44.584339 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:44.584342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:44.585285 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:44.585297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:44.586061 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:44.586066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:44.586192 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:44.586737 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:44.586754 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:44.589513 (+ 2759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":192,"threads_started":1}
W20260619 13:14:44.589902 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:44.582007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47159 (local address 127.0.97.62:37475)
0619 13:14:44.582259 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:44.582265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:44.582518 (+ 253us) server_negotiation.cc:407] Connection header received
0619 13:14:44.582672 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:44.582675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:44.582734 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:44.582810 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:44.583471 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:44.584235 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:44.585401 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:44.585941 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:44.586028 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:44.589682 (+ 3654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47159: BlockingRecv error: recv got EOF from 127.0.97.2:47159 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Jun 19 13:14:45 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:45 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:45.021203 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:45.013466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:43481)
0619 13:14:45.014034 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:45.014050 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:45.014166 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:45.014428 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:45.014431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:45.014807 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:45.014977 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:45.014986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:45.015975 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:45.015979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:45.016881 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:45.016893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:45.017571 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:45.017575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:45.017679 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:45.018284 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:45.018302 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:45.020955 (+ 2653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":230,"threads_started":1}
W20260619 13:14:45.021333 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:45.013533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43481 (local address 127.0.97.62:37475)
0619 13:14:45.013709 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:45.013714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:45.014116 (+ 402us) server_negotiation.cc:407] Connection header received
0619 13:14:45.014256 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:45.014260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:45.014317 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:45.014399 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:45.015128 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:45.015853 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:45.017019 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:45.017456 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:45.017555 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:45.021141 (+ 3586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43481: BlockingRecv error: recv got EOF from 127.0.97.3:43481 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 13:14:45 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:45 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:45.167043 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:45.159450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:36961)
0619 13:14:45.159974 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:45.159990 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:45.160103 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:45.160423 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:45.160428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:45.160906 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:45.161074 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:45.161084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:45.162012 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:45.162016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:45.162912 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:45.162923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:45.163556 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:45.163561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:45.163670 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:45.164190 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:45.164206 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:45.166822 (+ 2616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":309,"threads_started":1}
W20260619 13:14:45.167284 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:45.159570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36961 (local address 127.0.97.62:37475)
0619 13:14:45.159806 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:45.159812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:45.160061 (+ 249us) server_negotiation.cc:407] Connection header received
0619 13:14:45.160203 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:45.160206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:45.160270 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:45.160367 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:45.161189 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:45.161901 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:45.163017 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:45.163424 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:45.163511 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:45.167042 (+ 3531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36961: BlockingRecv error: recv got EOF from 127.0.97.1:36961 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 13:14:45 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:45 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:45.598654 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:45.590647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:33263)
0619 13:14:45.591087 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:45.591103 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:45.591228 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:45.591516 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:45.591520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:45.591916 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:45.592057 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:45.592064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:45.593264 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:45.593277 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:45.594524 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:45.594541 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:45.595208 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:45.595213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:45.595342 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:45.595857 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:45.595873 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:45.598441 (+ 2568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":230,"threads_started":1}
W20260619 13:14:45.598771 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:45.590745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33263 (local address 127.0.97.62:37475)
0619 13:14:45.590944 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:45.590950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:45.591182 (+ 232us) server_negotiation.cc:407] Connection header received
0619 13:14:45.591339 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:45.591345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:45.591402 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:45.591479 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:45.592189 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:45.593111 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:45.594655 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:45.595069 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:45.595163 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:45.598603 (+ 3440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33263: BlockingRecv error: recv got EOF from 127.0.97.2:33263 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 13:14:46 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:46 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:46.033936 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:46.022106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:49961)
0619 13:14:46.022513 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:46.022529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:46.022637 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:46.022944 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:46.022947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:46.023366 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:46.023513 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:46.023521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:46.024576 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:46.024580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:46.025467 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:46.025478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:46.026138 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:46.026143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:46.026266 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:46.030813 (+ 4547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:46.030833 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:46.033667 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":217,"threads_started":1}
W20260619 13:14:46.034076 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:46.022172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49961 (local address 127.0.97.62:37475)
0619 13:14:46.022405 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:46.022410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:46.022607 (+ 197us) server_negotiation.cc:407] Connection header received
0619 13:14:46.022727 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:46.022732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:46.022814 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:46.022895 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:46.023633 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:46.024456 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:46.025570 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:46.026014 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:46.026103 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:46.033884 (+ 7781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49961: BlockingRecv error: recv got EOF from 127.0.97.3:49961 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 13:14:46 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:46 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:46.176985 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:46.168007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:57551)
0619 13:14:46.168494 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:46.168513 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:46.168641 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:46.168985 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:46.168990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:46.169494 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:46.169699 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:46.169711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:46.171043 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:46.171048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:46.172215 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:46.172228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:46.172888 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:46.172893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:46.173056 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:46.173576 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:46.173598 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:46.176666 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":321,"threads_started":1}
W20260619 13:14:46.177085 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:46.168282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57551 (local address 127.0.97.62:37475)
0619 13:14:46.168459 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:46.168465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:46.168596 (+ 131us) server_negotiation.cc:407] Connection header received
0619 13:14:46.168758 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:46.168763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:46.168850 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:46.168951 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:46.169855 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:46.170880 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:46.172334 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:46.172758 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:46.172846 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:46.176875 (+ 4029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57551: BlockingRecv error: recv got EOF from 127.0.97.1:57551 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 13:14:46 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:46 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:46.607142 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:46.599426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:49707)
0619 13:14:46.599853 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:46.599868 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:46.599985 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:46.600281 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:46.600285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:46.600647 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:46.600807 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:46.600815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:46.601788 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:46.601793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:46.602726 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:46.602738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:46.603382 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:46.603387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:46.603543 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:46.604064 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:46.604080 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:46.606916 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":236,"threads_started":1}
W20260619 13:14:46.607265 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:46.599471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49707 (local address 127.0.97.62:37475)
0619 13:14:46.599628 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:46.599634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:46.599939 (+ 305us) server_negotiation.cc:407] Connection header received
0619 13:14:46.600087 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:46.600092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:46.600151 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:46.600226 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:46.600931 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:46.601664 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:46.602855 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:46.603253 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:46.603354 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:46.607079 (+ 3725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49707: BlockingRecv error: recv got EOF from 127.0.97.2:49707 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 13:14:47 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:47 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:47.044010 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:47.034888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:39353)
0619 13:14:47.035371 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:47.035386 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:47.035527 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:47.035883 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:47.035889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:47.036389 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:47.036615 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:47.036627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:47.037808 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:47.037812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:47.039094 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:47.039106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:47.039994 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:47.040000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:47.040170 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:47.040799 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:47.040823 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:47.043777 (+ 2954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":276,"threads_started":1}
W20260619 13:14:47.044152 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:47.034951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39353 (local address 127.0.97.62:37475)
0619 13:14:47.035121 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:47.035129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:47.035469 (+ 340us) server_negotiation.cc:407] Connection header received
0619 13:14:47.035651 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:47.035655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:47.035728 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:47.035829 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:47.036757 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:47.037696 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:47.039231 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:47.039825 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:47.039944 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:47.043974 (+ 4030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39353: BlockingRecv error: recv got EOF from 127.0.97.3:39353 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 13:14:47 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:47 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:47.185606 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:47.177956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:59103)
0619 13:14:47.178390 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:47.178406 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:47.178524 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:47.178821 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:47.178825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:47.179169 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:47.179310 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:47.179320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:47.180371 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:47.180375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:47.181295 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:47.181308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:47.182025 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:47.182030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:47.182184 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:47.182709 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:47.182725 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:47.185289 (+ 2564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":252,"threads_started":1}
W20260619 13:14:47.185703 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:47.178012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59103 (local address 127.0.97.62:37475)
0619 13:14:47.178194 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:47.178200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:47.178481 (+ 281us) server_negotiation.cc:407] Connection header received
0619 13:14:47.178625 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:47.178631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:47.178689 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:47.178765 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:47.179451 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:47.180227 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:47.181430 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:47.181892 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:47.181999 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:47.185484 (+ 3485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59103: BlockingRecv error: recv got EOF from 127.0.97.1:59103 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 13:14:47 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:47 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:47.615444 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:47.607957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:37921)
0619 13:14:47.608376 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:47.608391 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:47.608497 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:47.608796 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:47.608798 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:47.609140 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:47.609274 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:47.609282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:47.610214 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:47.610217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:47.611097 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:47.611109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:47.611725 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:47.611729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:47.611855 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:47.612407 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:47.612428 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:47.615165 (+ 2737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":238,"threads_started":1}
W20260619 13:14:47.615549 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:47.608022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37921 (local address 127.0.97.62:37475)
0619 13:14:47.608209 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:47.608216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:47.608533 (+ 317us) server_negotiation.cc:407] Connection header received
0619 13:14:47.608592 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:47.608596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:47.608659 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:47.608772 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:47.609396 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:47.610131 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:47.611216 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:47.611619 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:47.611707 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:47.615327 (+ 3620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37921: BlockingRecv error: recv got EOF from 127.0.97.2:37921 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 13:14:48 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:48 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:48.052205 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:48.044898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:44251)
0619 13:14:48.045376 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:48.045397 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:48.045535 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:48.045902 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:48.045907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:48.046286 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:48.046427 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:48.046436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:48.047384 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:48.047388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:48.048286 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:48.048296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:48.048869 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:48.048872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:48.048981 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:48.049469 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:48.049486 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:48.051995 (+ 2509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":299,"threads_started":1}
W20260619 13:14:48.052371 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:48.045039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44251 (local address 127.0.97.62:37475)
0619 13:14:48.045200 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:48.045206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:48.045480 (+ 274us) server_negotiation.cc:407] Connection header received
0619 13:14:48.045662 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:48.045668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:48.045742 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:48.045862 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:48.046547 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:48.047280 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:48.048385 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:48.048766 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:48.048851 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:48.052180 (+ 3329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44251: BlockingRecv error: recv got EOF from 127.0.97.3:44251 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 13:14:48 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:48 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:48.194217 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:48.186486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:60933)
0619 13:14:48.186927 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:48.186942 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:48.187067 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:48.187374 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:48.187379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:48.187761 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:48.187926 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:48.187935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:48.188912 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:48.188918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:48.189794 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:48.189806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:48.190490 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:48.190496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:48.190645 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:48.191208 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:48.191224 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:48.193962 (+ 2738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
W20260619 13:14:48.194407 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:48.186540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60933 (local address 127.0.97.62:37475)
0619 13:14:48.186758 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:48.186764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:48.187021 (+ 257us) server_negotiation.cc:407] Connection header received
0619 13:14:48.187185 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:48.187191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:48.187256 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:48.187334 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:48.188058 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:48.188789 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:48.189943 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:48.190352 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:48.190456 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:48.194169 (+ 3713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60933: BlockingRecv error: recv got EOF from 127.0.97.1:60933 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 13:14:48 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:48 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:48.624809 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:48.616480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:55717)
0619 13:14:48.616941 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:48.616958 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:48.617064 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:48.617365 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:48.617368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:48.617746 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:48.617964 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:48.617974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:48.619337 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:48.619341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:48.620220 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:48.620233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:48.621209 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:48.621215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:48.621356 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:48.621967 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:48.621986 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:48.624511 (+ 2525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":243,"threads_started":1}
W20260619 13:14:48.624913 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:48.616580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55717 (local address 127.0.97.62:37475)
0619 13:14:48.616746 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:48.616752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:48.617038 (+ 286us) server_negotiation.cc:407] Connection header received
0619 13:14:48.617165 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:48.617170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:48.617242 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:48.617355 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:48.618098 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:48.619187 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:48.620358 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:48.621070 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:48.621187 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:48.624702 (+ 3515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55717: BlockingRecv error: recv got EOF from 127.0.97.2:55717 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 13:14:49 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:49 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:49.060823 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:49.052959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:57741)
0619 13:14:49.053426 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:49.053441 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:49.053570 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:49.053893 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:49.053898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:49.054294 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:49.054456 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:49.054464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:49.055440 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:49.055445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:49.056330 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:49.056342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:49.057003 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:49.057009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:49.057169 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:49.057682 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:49.057698 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:49.060487 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":254,"threads_started":1}
W20260619 13:14:49.060912 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:49.053042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57741 (local address 127.0.97.62:37475)
0619 13:14:49.053243 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:49.053250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:49.053516 (+ 266us) server_negotiation.cc:407] Connection header received
0619 13:14:49.053682 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:49.053687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:49.053755 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:49.053831 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:49.054599 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:49.055316 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:49.056459 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:49.056865 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:49.056968 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:49.060700 (+ 3732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57741: BlockingRecv error: recv got EOF from 127.0.97.3:57741 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 13:14:49 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:49 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:49.202685 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:49.195014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:42837)
0619 13:14:49.195453 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:49.195469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:49.195576 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:49.195888 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:49.195892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:49.196292 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:49.196438 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:49.196448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:49.197529 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:49.197533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:49.198488 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:49.198500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:49.199184 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:49.199188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:49.199317 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:49.199824 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:49.199839 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:49.202387 (+ 2548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":250,"threads_started":1}
W20260619 13:14:49.202780 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:49.195092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42837 (local address 127.0.97.62:37475)
0619 13:14:49.195230 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:49.195234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:49.195583 (+ 349us) server_negotiation.cc:407] Connection header received
0619 13:14:49.195659 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:49.195663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:49.195737 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:49.195822 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:49.196614 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:49.197418 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:49.198633 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:49.199078 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:49.199177 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:49.202569 (+ 3392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42837: BlockingRecv error: recv got EOF from 127.0.97.1:42837 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 13:14:49 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:49 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:49.633240 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:49.625667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:38283)
0619 13:14:49.626153 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:49.626168 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:49.626292 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:49.626607 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:49.626612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:49.626984 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:49.627135 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:49.627144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:49.628069 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:49.628074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:49.628970 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:49.628981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:49.629655 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:49.629660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:49.629828 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:49.630431 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:49.630447 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:49.632940 (+ 2493us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":287,"threads_started":1}
W20260619 13:14:49.633327 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:49.625743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38283 (local address 127.0.97.62:37475)
0619 13:14:49.625961 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:49.625967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:49.626248 (+ 281us) server_negotiation.cc:407] Connection header received
0619 13:14:49.626401 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:49.626407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:49.626474 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:49.626550 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:49.627268 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:49.627940 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:49.629106 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:49.629524 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:49.629627 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:49.633122 (+ 3495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38283: BlockingRecv error: recv got EOF from 127.0.97.2:38283 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 13:14:50 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:50 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:50.070117 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:50.061702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:41383)
0619 13:14:50.062139 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:50.062155 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:50.062266 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:50.062537 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:50.062540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:50.062892 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:50.063037 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:50.063066 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:50.064402 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:50.064406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:50.065306 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:50.065318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:50.066468 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:50.066474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:50.066649 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:50.067286 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:50.067305 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:50.069880 (+ 2575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":250,"threads_started":1}
W20260619 13:14:50.070252 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:50.061768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41383 (local address 127.0.97.62:37475)
0619 13:14:50.061992 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:50.061998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:50.062223 (+ 225us) server_negotiation.cc:407] Connection header received
0619 13:14:50.062373 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:50.062377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:50.062434 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:50.062502 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:50.063183 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:50.064231 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:50.065428 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:50.066080 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:50.066197 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:50.070063 (+ 3866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41383: BlockingRecv error: recv got EOF from 127.0.97.3:41383 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 13:14:50 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:50 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:50.211225 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:50.203531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:41149)
0619 13:14:50.204036 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:50.204051 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:50.204178 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:50.204495 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:50.204499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:50.204874 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:50.205039 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:50.205048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:50.206128 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:50.206132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:50.207020 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:50.207033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:50.207734 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:50.207741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:50.207886 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:50.208398 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:50.208414 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:50.210931 (+ 2517us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":258,"threads_started":1}
W20260619 13:14:50.211333 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:50.203663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41149 (local address 127.0.97.62:37475)
0619 13:14:50.203879 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:50.203885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:50.204129 (+ 244us) server_negotiation.cc:407] Connection header received
0619 13:14:50.204284 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:50.204290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:50.204367 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:50.204446 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:50.205182 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:50.205995 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:50.207158 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:50.207572 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:50.207677 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:50.211115 (+ 3438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41149: BlockingRecv error: recv got EOF from 127.0.97.1:41149 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 13:14:50 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:50 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:50.642184 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:50.634149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:53535)
0619 13:14:50.634629 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:50.634645 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:50.634776 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:50.635145 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:50.635149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:50.635562 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:50.635801 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:50.635817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:50.636848 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:50.636852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:50.637957 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:50.637970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:50.638707 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:50.638713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:50.638876 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:50.639405 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:50.639422 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:50.641882 (+ 2460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":238,"threads_started":1}
W20260619 13:14:50.642294 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:50.634276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53535 (local address 127.0.97.62:37475)
0619 13:14:50.634421 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:50.634425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:50.634726 (+ 301us) server_negotiation.cc:407] Connection header received
0619 13:14:50.634907 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:50.634917 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:50.634992 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:50.635091 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:50.635930 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:50.636700 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:50.638087 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:50.638538 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:50.638644 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:50.642082 (+ 3438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53535: BlockingRecv error: recv got EOF from 127.0.97.2:53535 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 13:14:51 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:51 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:51.081790 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:51.070892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:52545)
0619 13:14:51.071329 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:51.071345 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:51.071455 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:51.071751 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:51.071754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:51.072124 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:51.072296 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:51.072305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:51.073289 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:51.073292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:51.074312 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:51.074325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:51.075009 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:51.075013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:51.075123 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:51.078864 (+ 3741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:51.078881 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:51.081568 (+ 2687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":210,"threads_started":1}
W20260619 13:14:51.081964 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:51.070981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52545 (local address 127.0.97.62:37475)
0619 13:14:51.071165 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:51.071170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:51.071412 (+ 242us) server_negotiation.cc:407] Connection header received
0619 13:14:51.071572 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:51.071578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:51.071644 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:51.071719 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:51.072412 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:51.073183 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:51.074457 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:51.074871 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:51.074970 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:51.081741 (+ 6771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52545: BlockingRecv error: recv got EOF from 127.0.97.3:52545 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 13:14:51 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:51 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:51.219545 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:51.212137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:58885)
0619 13:14:51.212553 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:51.212569 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:51.212680 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:51.212941 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:51.212944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:51.213269 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:51.213410 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:51.213418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:51.214471 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:51.214475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:51.215353 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:51.215363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:51.216020 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:51.216024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:51.216146 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:51.216703 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:51.216729 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:51.219298 (+ 2569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":202,"threads_started":1}
W20260619 13:14:51.219755 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:51.212241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58885 (local address 127.0.97.62:37475)
0619 13:14:51.212562 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:51.212567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:51.212634 (+ 67us) server_negotiation.cc:407] Connection header received
0619 13:14:51.212763 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:51.212766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:51.212837 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:51.212913 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:51.213554 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:51.214367 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:51.215480 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:51.215885 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:51.215981 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:51.219513 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58885: BlockingRecv error: recv got EOF from 127.0.97.1:58885 (error 108)
Metrics: {"server-negotiator.queue_time_us":209}
Jun 19 13:14:51 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:51 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:51.650342 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:51.643025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:34905)
0619 13:14:51.643454 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:51.643472 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:51.643582 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:51.643829 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:51.643832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:51.644191 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:51.644340 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:51.644349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:51.645328 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:51.645332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:51.646251 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:51.646262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:51.646892 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:51.646895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:51.647005 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:51.647571 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:51.647588 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:51.650103 (+ 2515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":228,"threads_started":1}
W20260619 13:14:51.650436 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:51.643113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34905 (local address 127.0.97.62:37475)
0619 13:14:51.643260 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:51.643264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:51.643533 (+ 269us) server_negotiation.cc:407] Connection header received
0619 13:14:51.643664 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:51.643667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:51.643723 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:51.643802 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:51.644489 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:51.645196 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:51.646371 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:51.646788 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:51.646877 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:51.650271 (+ 3394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34905: BlockingRecv error: recv got EOF from 127.0.97.2:34905 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 13:14:52 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:52 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:52.090380 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:52.082616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.3:40663)
0619 13:14:52.083027 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:52.083043 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:52.083155 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:52.083417 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:52.083420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:52.083754 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:52.083896 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:52.083904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.084907 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.084911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:52.086019 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:52.086032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.086693 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.086697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:52.086820 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:52.087390 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:52.087412 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:52.090132 (+ 2720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":221,"threads_started":1}
W20260619 13:14:52.090518 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:52.082690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40663 (local address 127.0.97.62:37475)
0619 13:14:52.082900 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:52.082907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:52.083111 (+ 204us) server_negotiation.cc:407] Connection header received
0619 13:14:52.083247 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:52.083250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:52.083309 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:52.083385 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:52.084010 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.084730 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.086137 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.086544 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.086639 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:52.090338 (+ 3699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40663: BlockingRecv error: recv got EOF from 127.0.97.3:40663 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 13:14:52 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:52 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:52.228438 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:52.220626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.1:41155)
0619 13:14:52.221057 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:52.221073 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:52.221181 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:52.221448 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:52.221451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:52.221819 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:52.222020 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:52.222030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.222966 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.222970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:52.223851 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:52.223861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.224548 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.224552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:52.224656 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:52.225195 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:52.225212 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:52.228106 (+ 2894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":230,"threads_started":1}
W20260619 13:14:52.228520 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:52.220684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41155 (local address 127.0.97.62:37475)
0619 13:14:52.220877 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:52.220884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:52.221169 (+ 285us) server_negotiation.cc:407] Connection header received
0619 13:14:52.221263 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:52.221266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:52.221341 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:52.221423 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:52.222168 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.222863 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.223974 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.224443 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.224533 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:52.228298 (+ 3765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41155: BlockingRecv error: recv got EOF from 127.0.97.1:41155 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 13:14:52 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 13:14:52 dist-test-slave-mkt3 krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 13:14:52.659071 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:52.651429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37475 (local address 127.0.97.2:50013)
0619 13:14:52.651925 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:52.651945 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:52.652070 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:52.652327 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:52.652330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:52.652702 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:52.652870 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:52.652878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.653819 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.653823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:52.654787 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:52.654801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.655493 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.655498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:52.655623 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:52.656201 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:52.656218 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:52.658820 (+ 2602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:37475: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"spinlock_wait_cycles":33024,"thread_start_us":345,"threads_started":1}
W20260619 13:14:52.659186 9343 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:52.651494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50013 (local address 127.0.97.62:37475)
0619 13:14:52.651690 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:52.651696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:52.652024 (+ 328us) server_negotiation.cc:407] Connection header received
0619 13:14:52.652158 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:52.652162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:52.652220 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:52.652311 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:52.652970 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.653682 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.654911 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.655349 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.655440 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:52.658991 (+ 3551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50013: BlockingRecv error: recv got EOF from 127.0.97.2:50013 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:14:52.932263 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 9194
I20260619 13:14:52.943529 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:52.933349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.0.97.1:43621)
0619 13:14:52.933889 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:52.933897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:52.933923 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:14:52.933988 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:52.933995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:52.934203 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:52.934353 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:52.935687 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.936715 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.937937 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.938457 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.938554 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:52.940740 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:52.940754 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:52.940768 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:52.940801 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:52.943088 (+ 2287us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 13:14:52.943313 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45308: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":270,"threads_started":1}
W20260619 13:14:52.943626 9654 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45308: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 13:14:52.943691 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 9194: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:43621: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 13:14:52.943892 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 9194
I20260619 13:14:52.958441 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 9345
I20260619 13:14:52.969303 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:52.959353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59262 (local address 127.0.97.2:40367)
0619 13:14:52.959845 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:52.959852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:52.959881 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:14:52.959937 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:52.959944 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:52.960156 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:52.960304 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:52.961455 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.962521 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.963633 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.964160 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.964255 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:52.966453 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:52.966462 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:52.966476 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:52.966505 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:52.968827 (+ 2322us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 13:14:52.969071 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59262: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":213,"threads_started":1}
W20260619 13:14:52.969401 9655 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59262: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 13:14:52.969472 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 9345: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:40367: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 13:14:52.969637 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 9345
I20260619 13:14:52.982497 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 9476
I20260619 13:14:52.993397 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:52.983311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.0.97.3:34071)
0619 13:14:52.983840 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:52.983848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:52.983872 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:14:52.983927 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:52.983933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:52.984140 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:52.984300 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:52.985607 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.986693 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.987789 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:52.988325 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:52.988412 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:52.990782 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:52.990790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:52.990803 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:52.990833 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:52.992927 (+ 2094us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 13:14:52.993150 (+ 223us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48268: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":269,"threads_started":1}
W20260619 13:14:52.993500 9656 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48268: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 13:14:52.993501 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 9476: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:34071: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 13:14:52.993687 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 9476
I20260619 13:14:53.006336 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 9125
I20260619 13:14:53.007586 9187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:53.117857 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 9125
2026-06-19T13:14:53Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20587 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T13:14:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:14:53Z Disabled control of system clock
I20260619 13:14:53.184146 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34511
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 13:14:53.489485 9661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:53.489761 9661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:53.489888 9661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:53.496878 9661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:53.496974 9661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:53.497010 9661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:53.497063 9661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:53.497112 9661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:14:53.505996 9661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:53.507174 9661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:53.508852 9661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:53.519424 9669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:53.519603 9666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:53.519603 9667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:53.520397 9661 server_base.cc:1034] running on GCE node
I20260619 13:14:53.521124 9661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:53.522389 9661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:53.523578 9661 hybrid_clock.cc:648] HybridClock initialized: now 1781874893523537 us; error 32 us; skew 500 ppm
I20260619 13:14:53.523857 9661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:53.525817 9661 webserver.cc:460] Webserver started at http://127.0.97.62:42149/ using document root <none> and password file <none>
I20260619 13:14:53.526464 9661 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:53.526543 9661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:53.526795 9661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:53.529371 9661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "300346b05b214168b8d9d9f962230e9d"
format_stamp: "Formatted at 2026-06-19 13:14:53 on dist-test-slave-mkt3"
server_key: "e037b5f0ad94b5d9d31acb6bebcb6ced"
server_key_iv: "5005f46bd4b9ee8d199cb725edd78a99"
server_key_version: "encryptionkey@0"
I20260619 13:14:53.530093 9661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "300346b05b214168b8d9d9f962230e9d"
format_stamp: "Formatted at 2026-06-19 13:14:53 on dist-test-slave-mkt3"
server_key: "e037b5f0ad94b5d9d31acb6bebcb6ced"
server_key_iv: "5005f46bd4b9ee8d199cb725edd78a99"
server_key_version: "encryptionkey@0"
I20260619 13:14:53.534638 9661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 13:14:53.537563 9675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:53.539044 9661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:14:53.539186 9661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "300346b05b214168b8d9d9f962230e9d"
format_stamp: "Formatted at 2026-06-19 13:14:53 on dist-test-slave-mkt3"
server_key: "e037b5f0ad94b5d9d31acb6bebcb6ced"
server_key_iv: "5005f46bd4b9ee8d199cb725edd78a99"
server_key_version: "encryptionkey@0"
I20260619 13:14:53.539436 9661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:53.585623 9661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:53.586454 9661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:53.586704 9661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:53.602602 9661 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34511
I20260619 13:14:53.602627 9726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34511 every 8 connection(s)
I20260619 13:14:53.603796 9661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:14:53.607340 9727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:53.612370 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 9661
I20260619 13:14:53.612537 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:14:53.613157 388 external_mini_cluster.cc:1428] Setting key ca1d9fda87be9ff3f930e141c1e146c7
I20260619 13:14:53.617213 9727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d: Bootstrap starting.
I20260619 13:14:53.620307 9727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:53.621469 9727 log.cc:824] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:53.624384 9727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d: No bootstrap required, opened a new log
I20260619 13:14:53.626228 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:53.615059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.0.97.62:34511)
0619 13:14:53.615969 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:53.615979 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:53.616018 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:14:53.617554 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:53.617576 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:53.618053 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:53.618353 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:53.618964 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:53.620174 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:53.621932 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:53.622891 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:53.623011 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:53.624099 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:53.624106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:53.624118 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:53.624659 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:53.624719 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:53.624745 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:53.624839 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:53.625033 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 13:14:53.625300 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":319,"threads_started":1}
I20260619 13:14:53.632025 9727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300346b05b214168b8d9d9f962230e9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34511 } }
I20260619 13:14:53.632323 9727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:53.632402 9727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 300346b05b214168b8d9d9f962230e9d, State: Initialized, Role: FOLLOWER
I20260619 13:14:53.632866 9727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300346b05b214168b8d9d9f962230e9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34511 } }
I20260619 13:14:53.633057 9727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:14:53.633167 9727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:14:53.633325 9727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:53.635313 9727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300346b05b214168b8d9d9f962230e9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34511 } }
I20260619 13:14:53.635727 9727 leader_election.cc:302] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 300346b05b214168b8d9d9f962230e9d; no voters:
I20260619 13:14:53.636260 9727 leader_election.cc:288] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:14:53.636368 9732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:53.636978 9732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [term 1 LEADER]: Becoming Leader. State: Replica: 300346b05b214168b8d9d9f962230e9d, State: Running, Role: LEADER
I20260619 13:14:53.637490 9732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300346b05b214168b8d9d9f962230e9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34511 } }
I20260619 13:14:53.638041 9727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:53.640105 9733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "300346b05b214168b8d9d9f962230e9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300346b05b214168b8d9d9f962230e9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34511 } } }
I20260619 13:14:53.640440 9733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:53.640184 9734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 300346b05b214168b8d9d9f962230e9d. Latest consensus state: current_term: 1 leader_uuid: "300346b05b214168b8d9d9f962230e9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "300346b05b214168b8d9d9f962230e9d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34511 } } }
I20260619 13:14:53.640674 9734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:53.646119 9738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:14:53.650944 9738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:14:53.658980 9738 catalog_manager.cc:1347] Generated new cluster ID: 61efc9d108794f8d982cb4e489c4be0f
I20260619 13:14:53.659071 9738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:14:53.665941 9738 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:14:53.667002 9738 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:14:53.677769 9738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 300346b05b214168b8d9d9f962230e9d: Generated new TSK 0
I20260619 13:14:53.678588 9738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:14:53.692870 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34511
--builtin_ntp_servers=127.0.97.20:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 13:14:53.999439 9751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:53.999720 9751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:53.999826 9751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:54.006953 9751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:54.007043 9751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:54.007177 9751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:14:54.015868 9751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:54.017030 9751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:54.018710 9751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:54.030247 9756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:54.030221 9757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:54.030472 9759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:54.031038 9751 server_base.cc:1034] running on GCE node
I20260619 13:14:54.031576 9751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:54.032433 9751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:54.033623 9751 hybrid_clock.cc:648] HybridClock initialized: now 1781874894033581 us; error 37 us; skew 500 ppm
I20260619 13:14:54.033953 9751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:54.035991 9751 webserver.cc:460] Webserver started at http://127.0.97.1:46163/ using document root <none> and password file <none>
I20260619 13:14:54.036602 9751 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:54.036705 9751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:54.037010 9751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:54.039701 9751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "2ea0f4e8e86442bf9bb4d39fb2dff389"
format_stamp: "Formatted at 2026-06-19 13:14:54 on dist-test-slave-mkt3"
server_key: "62e44884128df6a1d7c433e461730947"
server_key_iv: "380a58a7604d39fc7c8ad2c240057928"
server_key_version: "encryptionkey@0"
I20260619 13:14:54.040388 9751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2ea0f4e8e86442bf9bb4d39fb2dff389"
format_stamp: "Formatted at 2026-06-19 13:14:54 on dist-test-slave-mkt3"
server_key: "62e44884128df6a1d7c433e461730947"
server_key_iv: "380a58a7604d39fc7c8ad2c240057928"
server_key_version: "encryptionkey@0"
I20260619 13:14:54.045083 9751 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:14:54.048286 9765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:54.049484 9751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 13:14:54.049616 9751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "2ea0f4e8e86442bf9bb4d39fb2dff389"
format_stamp: "Formatted at 2026-06-19 13:14:54 on dist-test-slave-mkt3"
server_key: "62e44884128df6a1d7c433e461730947"
server_key_iv: "380a58a7604d39fc7c8ad2c240057928"
server_key_version: "encryptionkey@0"
I20260619 13:14:54.049801 9751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:54.068111 9751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:54.068858 9751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:54.069098 9751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:54.069984 9751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:54.071303 9751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:54.071395 9751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:54.071491 9751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:54.071558 9751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:54.099807 9751 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44247
I20260619 13:14:54.099854 9877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44247 every 8 connection(s)
I20260619 13:14:54.101003 9751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:14:54.104059 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 9751
I20260619 13:14:54.104233 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:14:54.104846 388 external_mini_cluster.cc:1428] Setting key 48ce62ae38a7dc8bfdee19ce4b59236d
I20260619 13:14:54.107421 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34511
--builtin_ntp_servers=127.0.97.20:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 13:14:54.113260 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:54.103481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47437 (local address 127.0.97.62:34511)
0619 13:14:54.103686 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:54.103692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:54.105129 (+ 1437us) server_negotiation.cc:407] Connection header received
0619 13:14:54.106950 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:54.106956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:54.107028 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:54.107136 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:54.108142 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:54.109106 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:54.110762 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:54.111269 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:54.111377 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:54.112558 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:54.112564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:54.112575 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:54.112602 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:54.112636 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:54.112649 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:54.112724 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:54.112890 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 13:14:54.112973 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:14:54.114372 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:54.103993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34511 (local address 127.0.97.1:47437)
0619 13:14:54.104982 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:54.105023 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:54.106714 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:54.107321 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:54.107329 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:54.107412 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:54.107991 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:54.108007 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:54.109298 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:54.109302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:54.110635 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:54.110650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:54.111458 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:54.111463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:54.111673 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:54.112290 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:14:54.112293 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:54.112330 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:14:54.112400 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:54.112731 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:54.112737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:54.112742 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:14:54.112867 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 13:14:54.113143 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":308,"threads_started":1}
I20260619 13:14:54.115896 9878 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34511
I20260619 13:14:54.116151 9878 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:54.116855 9878 heartbeater.cc:511] Master 127.0.97.62:34511 requested a full tablet report, sending...
I20260619 13:14:54.118643 9692 ts_manager.cc:194] Registered new tserver with Master: 2ea0f4e8e86442bf9bb4d39fb2dff389 (127.0.97.1:44247)
I20260619 13:14:54.120056 9692 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:47437
W20260619 13:14:54.414253 9881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:54.414554 9881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:54.414692 9881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:54.421767 9881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:54.421895 9881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:54.422041 9881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:14:54.431174 9881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:54.432381 9881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:54.434082 9881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:54.446585 9886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:54.446656 9887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:54.446610 9889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:54.447304 9881 server_base.cc:1034] running on GCE node
I20260619 13:14:54.447924 9881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:54.448805 9881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:54.449986 9881 hybrid_clock.cc:648] HybridClock initialized: now 1781874894449922 us; error 71 us; skew 500 ppm
I20260619 13:14:54.450284 9881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:54.452346 9881 webserver.cc:460] Webserver started at http://127.0.97.2:43173/ using document root <none> and password file <none>
I20260619 13:14:54.453030 9881 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:54.453127 9881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:54.453451 9881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:54.456239 9881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "467c05f102a24e1cb49f263a756b4864"
format_stamp: "Formatted at 2026-06-19 13:14:54 on dist-test-slave-mkt3"
server_key: "28c18f3d5878455e421d1b14c15c8a7d"
server_key_iv: "229659e0a58dfe25aea01b6788063d2d"
server_key_version: "encryptionkey@0"
I20260619 13:14:54.456981 9881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "467c05f102a24e1cb49f263a756b4864"
format_stamp: "Formatted at 2026-06-19 13:14:54 on dist-test-slave-mkt3"
server_key: "28c18f3d5878455e421d1b14c15c8a7d"
server_key_iv: "229659e0a58dfe25aea01b6788063d2d"
server_key_version: "encryptionkey@0"
I20260619 13:14:54.461781 9881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:14:54.464917 9895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:54.466205 9881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:14:54.466362 9881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "467c05f102a24e1cb49f263a756b4864"
format_stamp: "Formatted at 2026-06-19 13:14:54 on dist-test-slave-mkt3"
server_key: "28c18f3d5878455e421d1b14c15c8a7d"
server_key_iv: "229659e0a58dfe25aea01b6788063d2d"
server_key_version: "encryptionkey@0"
I20260619 13:14:54.466539 9881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:54.484581 9881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:54.485289 9881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:54.485517 9881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:54.486393 9881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:54.487666 9881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:54.487761 9881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:54.487854 9881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:54.487921 9881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:54.516525 9881 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42001
I20260619 13:14:54.516568 10007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42001 every 8 connection(s)
I20260619 13:14:54.517735 9881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:14:54.527303 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 9881
I20260619 13:14:54.527469 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:14:54.528009 388 external_mini_cluster.cc:1428] Setting key 02eba51772526f746837313eeb76a057
I20260619 13:14:54.529302 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:54.520132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58509 (local address 127.0.97.62:34511)
0619 13:14:54.520298 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:54.520302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:54.521629 (+ 1327us) server_negotiation.cc:407] Connection header received
0619 13:14:54.523325 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:54.523329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:54.523393 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:54.523484 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:54.524454 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:54.525185 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:54.526656 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:54.527061 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:54.527159 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:54.528587 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:54.528592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:54.528601 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:54.528619 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:54.528656 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:54.528670 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:54.528763 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:54.528981 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 13:14:54.529056 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:14:54.530509 10010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:54.520617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34511 (local address 127.0.97.2:58509)
0619 13:14:54.521475 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:54.521514 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:54.523071 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:54.523673 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:54.523681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:54.523766 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:54.524307 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:54.524324 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:54.525291 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:54.525294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:54.526536 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:54.526547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:54.527272 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:54.527279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:54.527559 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:54.528296 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:14:54.528298 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:54.528336 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:14:54.528415 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:54.528766 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:54.528781 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:54.528788 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 13:14:54.528958 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 13:14:54.529335 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":246,"threads_started":1}
I20260619 13:14:54.530829 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34511
--builtin_ntp_servers=127.0.97.20:33615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 13:14:54.531757 10008 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34511
I20260619 13:14:54.532008 10008 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:54.532692 10008 heartbeater.cc:511] Master 127.0.97.62:34511 requested a full tablet report, sending...
I20260619 13:14:54.533960 9692 ts_manager.cc:194] Registered new tserver with Master: 467c05f102a24e1cb49f263a756b4864 (127.0.97.2:42001)
I20260619 13:14:54.534739 9692 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:58509
W20260619 13:14:54.833982 10011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:54.834237 10011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:54.834350 10011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:54.841306 10011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:54.841392 10011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:54.841517 10011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:14:54.850309 10011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:54.851466 10011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:54.853125 10011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:54.864873 10016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:54.864916 10017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:54.864987 10019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:54.865484 10011 server_base.cc:1034] running on GCE node
I20260619 13:14:54.866113 10011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:54.866918 10011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:54.868119 10011 hybrid_clock.cc:648] HybridClock initialized: now 1781874894868065 us; error 51 us; skew 500 ppm
I20260619 13:14:54.868387 10011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:54.870431 10011 webserver.cc:460] Webserver started at http://127.0.97.3:40327/ using document root <none> and password file <none>
I20260619 13:14:54.871038 10011 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:54.871142 10011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:54.871428 10011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:54.874015 10011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "13f87c8151db444aba01be8ce143285e"
format_stamp: "Formatted at 2026-06-19 13:14:54 on dist-test-slave-mkt3"
server_key: "110ddb3780ae0e823fd98680f0a3cd1c"
server_key_iv: "1588efec08d51cf454094b9e8a69437d"
server_key_version: "encryptionkey@0"
I20260619 13:14:54.874729 10011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "13f87c8151db444aba01be8ce143285e"
format_stamp: "Formatted at 2026-06-19 13:14:54 on dist-test-slave-mkt3"
server_key: "110ddb3780ae0e823fd98680f0a3cd1c"
server_key_iv: "1588efec08d51cf454094b9e8a69437d"
server_key_version: "encryptionkey@0"
I20260619 13:14:54.879366 10011 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 13:14:54.882427 10025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:54.883607 10011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 13:14:54.883746 10011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "13f87c8151db444aba01be8ce143285e"
format_stamp: "Formatted at 2026-06-19 13:14:54 on dist-test-slave-mkt3"
server_key: "110ddb3780ae0e823fd98680f0a3cd1c"
server_key_iv: "1588efec08d51cf454094b9e8a69437d"
server_key_version: "encryptionkey@0"
I20260619 13:14:54.883919 10011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:54.900712 10011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:54.901487 10011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:54.901729 10011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:54.902614 10011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:54.903983 10011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:54.904083 10011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:54.904187 10011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:54.904254 10011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:54.932184 10011 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41457
I20260619 13:14:54.932231 10137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41457 every 8 connection(s)
I20260619 13:14:54.933387 10011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:14:54.939810 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 10011
I20260619 13:14:54.939935 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:14:54.940363 388 external_mini_cluster.cc:1428] Setting key 3b27f11daa8424a815f3acaada89e736
I20260619 13:14:54.945254 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:54.935715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36779 (local address 127.0.97.62:34511)
0619 13:14:54.935877 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:54.935881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:54.937188 (+ 1307us) server_negotiation.cc:407] Connection header received
0619 13:14:54.938920 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:54.938925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:54.938997 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:54.939079 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:54.940143 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:54.940896 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:54.942859 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:54.943305 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:54.943392 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:54.944634 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:54.944638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:54.944641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:54.944655 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:54.944682 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:54.944692 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:54.944757 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:54.944952 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 13:14:54.945022 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:14:54.946180 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:54.936209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34511 (local address 127.0.97.3:36779)
0619 13:14:54.937039 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:54.937078 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:54.938670 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:54.939306 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:54.939315 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:54.939395 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:54.939988 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:54.940013 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:54.941070 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:54.941075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:54.942731 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:54.942753 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:54.943489 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:54.943494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:54.943721 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:54.944381 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:14:54.944383 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:54.944419 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:14:54.944489 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:54.944756 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:54.944762 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:54.944767 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:14:54.944893 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 13:14:54.945157 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":245,"threads_started":1}
I20260619 13:14:54.947342 10138 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34511
I20260619 13:14:54.947626 10138 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:54.948462 10138 heartbeater.cc:511] Master 127.0.97.62:34511 requested a full tablet report, sending...
I20260619 13:14:54.949880 9692 ts_manager.cc:194] Registered new tserver with Master: 13f87c8151db444aba01be8ce143285e (127.0.97.3:41457)
I20260619 13:14:54.950857 9692 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:36779
I20260619 13:14:54.954892 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 13:14:54.960847 9730 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:54.959138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46100 (local address 127.0.97.62:34511)
0619 13:14:54.959290 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:54.959296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:54.959745 (+ 449us) server_negotiation.cc:407] Connection header received
0619 13:14:54.959903 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:54.959908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:54.959983 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:54.960061 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:54.960600 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46100: BlockingRecv error: recv got EOF from 127.0.0.1:46100 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:14:54.961442 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 9751
I20260619 13:14:54.968649 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:54.962272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55098 (local address 127.0.97.1:44247)
0619 13:14:54.962728 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:54.962734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:54.962758 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:14:54.962812 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:54.962818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:54.963006 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:54.963159 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:54.963502 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:54.964489 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:54.965660 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:54.966253 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:54.966365 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:54.967326 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:54.967333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:54.967345 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:54.967909 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:54.967953 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:54.967975 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:54.968081 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:54.968212 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 13:14:54.968348 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":206,"threads_started":1}
I20260619 13:14:54.969787 9873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:55.048420 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 9751
I20260619 13:14:55.061653 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 9881
I20260619 13:14:55.068929 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:55.062506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46076 (local address 127.0.97.2:42001)
0619 13:14:55.062983 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:55.062990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:55.063015 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:14:55.063071 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:55.063078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:55.063285 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:55.063408 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:55.063733 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:55.064693 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:55.065790 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:55.066408 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:55.066498 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:55.067408 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:55.067415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:55.067427 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:55.068035 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:55.068092 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:55.068116 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:55.068220 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:55.068423 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 13:14:55.068562 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":244,"threads_started":1}
I20260619 13:14:55.070398 10003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:55.149093 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 9881
I20260619 13:14:55.161590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 10011
I20260619 13:14:55.169145 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:55.162585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.0.97.3:41457)
0619 13:14:55.163242 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:55.163249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:55.163279 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:14:55.163341 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:55.163347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:55.163558 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:55.163679 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:55.164035 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:55.165048 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:55.166261 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:55.166799 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:55.166892 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:55.167861 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:55.167869 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:55.167883 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:55.168454 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:55.168498 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:55.168517 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:55.168601 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:55.168692 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 13:14:55.168804 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":342,"threads_started":1}
I20260619 13:14:55.170368 10133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:55.250073 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 10011
I20260619 13:14:55.262955 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 9661
I20260619 13:14:55.264051 9722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:55.359592 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 9661
2026-06-19T13:14:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2236 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T13:14:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:14:55Z Disabled control of system clock
I20260619 13:14:55.421479 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38125
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 13:14:55.724200 10157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:55.724471 10157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:55.724560 10157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:55.731630 10157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:55.731734 10157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:55.731777 10157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:55.731830 10157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:55.731873 10157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:14:55.740697 10157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38125
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:55.741894 10157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:55.743583 10157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:55.754400 10162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:55.754369 10163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:55.754439 10165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:55.755252 10157 server_base.cc:1034] running on GCE node
I20260619 13:14:55.755990 10157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:55.757341 10157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:55.758541 10157 hybrid_clock.cc:648] HybridClock initialized: now 1781874895758487 us; error 50 us; skew 500 ppm
I20260619 13:14:55.758868 10157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:55.760914 10157 webserver.cc:460] Webserver started at http://127.0.97.62:45613/ using document root <none> and password file <none>
I20260619 13:14:55.761564 10157 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:55.761665 10157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:55.762012 10157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:55.764714 10157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "4532095168e44bf9a52430bdfc8334ff"
format_stamp: "Formatted at 2026-06-19 13:14:55 on dist-test-slave-mkt3"
server_key: "46a6cddb28c7b123768ce0df1629fb5b"
server_key_iv: "b94f556b7bcfeee992395aa5329f4e2e"
server_key_version: "encryptionkey@0"
I20260619 13:14:55.765456 10157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "4532095168e44bf9a52430bdfc8334ff"
format_stamp: "Formatted at 2026-06-19 13:14:55 on dist-test-slave-mkt3"
server_key: "46a6cddb28c7b123768ce0df1629fb5b"
server_key_iv: "b94f556b7bcfeee992395aa5329f4e2e"
server_key_version: "encryptionkey@0"
I20260619 13:14:55.770311 10157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:14:55.773452 10171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:55.774712 10157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:14:55.774858 10157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "4532095168e44bf9a52430bdfc8334ff"
format_stamp: "Formatted at 2026-06-19 13:14:55 on dist-test-slave-mkt3"
server_key: "46a6cddb28c7b123768ce0df1629fb5b"
server_key_iv: "b94f556b7bcfeee992395aa5329f4e2e"
server_key_version: "encryptionkey@0"
I20260619 13:14:55.775043 10157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:55.780203 10157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:55.780921 10157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:55.781221 10157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:55.796722 10157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38125
I20260619 13:14:55.796756 10222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38125 every 8 connection(s)
I20260619 13:14:55.797940 10157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:14:55.798401 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 10157
I20260619 13:14:55.798581 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:14:55.799309 388 external_mini_cluster.cc:1428] Setting key 6c8ce7f102ed9b095ca6caf53c03d171
I20260619 13:14:55.801919 10223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:55.807394 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:55.801220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.0.97.62:38125)
0619 13:14:55.802227 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:55.802238 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:55.802287 (+ 49us) server_negotiation.cc:407] Connection header received
0619 13:14:55.803761 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:55.803782 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:55.804197 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:55.804595 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:55.805730 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:55.805736 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:55.805753 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:55.805767 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:55.805874 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:55.805898 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:55.806103 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:55.806241 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 13:14:55.806496 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":330,"threads_started":1}
I20260619 13:14:55.812597 10223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff: Bootstrap starting.
I20260619 13:14:55.815363 10223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:55.816401 10223 log.cc:824] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:55.818825 10223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff: No bootstrap required, opened a new log
I20260619 13:14:55.825513 10223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4532095168e44bf9a52430bdfc8334ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38125 } }
I20260619 13:14:55.825826 10223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:55.825937 10223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4532095168e44bf9a52430bdfc8334ff, State: Initialized, Role: FOLLOWER
I20260619 13:14:55.826427 10223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4532095168e44bf9a52430bdfc8334ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38125 } }
I20260619 13:14:55.826625 10223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:14:55.826753 10223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:14:55.826900 10223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:55.828926 10223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4532095168e44bf9a52430bdfc8334ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38125 } }
I20260619 13:14:55.829344 10223 leader_election.cc:302] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4532095168e44bf9a52430bdfc8334ff; no voters:
I20260619 13:14:55.829818 10223 leader_election.cc:288] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:14:55.829964 10228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:55.830564 10228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [term 1 LEADER]: Becoming Leader. State: Replica: 4532095168e44bf9a52430bdfc8334ff, State: Running, Role: LEADER
I20260619 13:14:55.831043 10228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4532095168e44bf9a52430bdfc8334ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38125 } }
I20260619 13:14:55.831661 10223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:55.833931 10230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4532095168e44bf9a52430bdfc8334ff. Latest consensus state: current_term: 1 leader_uuid: "4532095168e44bf9a52430bdfc8334ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4532095168e44bf9a52430bdfc8334ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38125 } } }
I20260619 13:14:55.833909 10229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4532095168e44bf9a52430bdfc8334ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4532095168e44bf9a52430bdfc8334ff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38125 } } }
I20260619 13:14:55.834293 10230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:55.834384 10229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:55.835832 10235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:14:55.840451 10235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:14:55.848610 10235 catalog_manager.cc:1347] Generated new cluster ID: 63ae1f88fbf845c682c92de679f64814
I20260619 13:14:55.848708 10235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:14:55.867620 10235 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:14:55.868055 10235 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:14:55.881140 10235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4532095168e44bf9a52430bdfc8334ff: Generated new TSK 0
I20260619 13:14:55.882200 10235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:14:55.888757 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38125
--builtin_ntp_servers=127.0.97.20:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 13:14:56.203951 10247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:56.204217 10247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:56.204303 10247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:56.211499 10247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:56.211602 10247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:56.211709 10247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:14:56.220562 10247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:56.221751 10247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:56.223516 10247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:56.235733 10255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:56.235781 10253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:56.235807 10252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:56.236529 10247 server_base.cc:1034] running on GCE node
I20260619 13:14:56.237048 10247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:56.237898 10247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:56.239084 10247 hybrid_clock.cc:648] HybridClock initialized: now 1781874896239037 us; error 44 us; skew 500 ppm
I20260619 13:14:56.239373 10247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:56.244000 10247 webserver.cc:460] Webserver started at http://127.0.97.1:42411/ using document root <none> and password file <none>
I20260619 13:14:56.244647 10247 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:56.244731 10247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:56.245020 10247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:56.247763 10247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "5ef1c0c8f34d4eac8b475f935b885dbc"
format_stamp: "Formatted at 2026-06-19 13:14:56 on dist-test-slave-mkt3"
server_key: "e9de4247902e62eded78a5be74e77353"
server_key_iv: "c628db7e358b15efc6144cf6ccd87105"
server_key_version: "encryptionkey@0"
I20260619 13:14:56.248478 10247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5ef1c0c8f34d4eac8b475f935b885dbc"
format_stamp: "Formatted at 2026-06-19 13:14:56 on dist-test-slave-mkt3"
server_key: "e9de4247902e62eded78a5be74e77353"
server_key_iv: "c628db7e358b15efc6144cf6ccd87105"
server_key_version: "encryptionkey@0"
I20260619 13:14:56.253278 10247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260619 13:14:56.256703 10261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:56.258117 10247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:14:56.258284 10247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "5ef1c0c8f34d4eac8b475f935b885dbc"
format_stamp: "Formatted at 2026-06-19 13:14:56 on dist-test-slave-mkt3"
server_key: "e9de4247902e62eded78a5be74e77353"
server_key_iv: "c628db7e358b15efc6144cf6ccd87105"
server_key_version: "encryptionkey@0"
I20260619 13:14:56.258496 10247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:56.263935 10247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:56.264699 10247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:56.264933 10247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:56.265801 10247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:56.267336 10247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:56.267446 10247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:56.267561 10247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:56.267634 10247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:56.295467 10247 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41505
I20260619 13:14:56.295533 10373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41505 every 8 connection(s)
I20260619 13:14:56.296710 10247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:14:56.298041 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 10247
I20260619 13:14:56.298234 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:14:56.298933 388 external_mini_cluster.cc:1428] Setting key c3f4686dba0448c7c7528f945ecd5979
I20260619 13:14:56.302011 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38125
--builtin_ntp_servers=127.0.97.20:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 13:14:56.305351 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:56.299340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51835 (local address 127.0.97.62:38125)
0619 13:14:56.299494 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:56.299500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:56.301064 (+ 1564us) server_negotiation.cc:407] Connection header received
0619 13:14:56.303271 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:56.303275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:56.303321 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:56.303401 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:56.304733 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:56.304738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:56.304741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:56.304755 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:56.304788 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:56.304799 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:56.304882 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:56.305096 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 13:14:56.305146 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 13:14:56.306491 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:56.299912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38125 (local address 127.0.97.1:51835)
0619 13:14:56.300896 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:56.300942 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:56.302990 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:56.303654 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:56.303666 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:56.303775 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:56.304414 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:14:56.304418 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:56.304463 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:14:56.304561 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:56.304877 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:56.304884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:56.304891 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 13:14:56.305127 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 13:14:56.305360 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":291,"threads_started":1}
I20260619 13:14:56.307816 10374 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38125
I20260619 13:14:56.308107 10374 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:56.308732 10374 heartbeater.cc:511] Master 127.0.97.62:38125 requested a full tablet report, sending...
I20260619 13:14:56.310936 10188 ts_manager.cc:194] Registered new tserver with Master: 5ef1c0c8f34d4eac8b475f935b885dbc (127.0.97.1:41505)
W20260619 13:14:56.610370 10377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:56.610641 10377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:56.610728 10377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:56.617946 10377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:56.618068 10377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:56.618181 10377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:14:56.627157 10377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:56.628393 10377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:56.630112 10377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:56.642349 10383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:56.642388 10382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:56.642311 10385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:56.643276 10377 server_base.cc:1034] running on GCE node
I20260619 13:14:56.643841 10377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:56.644678 10377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:56.645900 10377 hybrid_clock.cc:648] HybridClock initialized: now 1781874896645848 us; error 43 us; skew 500 ppm
I20260619 13:14:56.646188 10377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:56.648180 10377 webserver.cc:460] Webserver started at http://127.0.97.2:41133/ using document root <none> and password file <none>
I20260619 13:14:56.648811 10377 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:56.648906 10377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:56.649186 10377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:56.653918 10377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "ec1c33eaa72e4574b4b246d12de1fe92"
format_stamp: "Formatted at 2026-06-19 13:14:56 on dist-test-slave-mkt3"
server_key: "936ba978bacd6057f346e414daca84e7"
server_key_iv: "2dc4589b99197128052f7c85f0514a9d"
server_key_version: "encryptionkey@0"
I20260619 13:14:56.654781 10377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ec1c33eaa72e4574b4b246d12de1fe92"
format_stamp: "Formatted at 2026-06-19 13:14:56 on dist-test-slave-mkt3"
server_key: "936ba978bacd6057f346e414daca84e7"
server_key_iv: "2dc4589b99197128052f7c85f0514a9d"
server_key_version: "encryptionkey@0"
I20260619 13:14:56.659677 10377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:14:56.663029 10391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:56.664336 10377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:14:56.664462 10377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "ec1c33eaa72e4574b4b246d12de1fe92"
format_stamp: "Formatted at 2026-06-19 13:14:56 on dist-test-slave-mkt3"
server_key: "936ba978bacd6057f346e414daca84e7"
server_key_iv: "2dc4589b99197128052f7c85f0514a9d"
server_key_version: "encryptionkey@0"
I20260619 13:14:56.664659 10377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:56.670022 10377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:56.670724 10377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:56.670928 10377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:56.671715 10377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:56.672917 10377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:56.672998 10377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:56.673074 10377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:56.673125 10377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:56.700989 10377 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41125
I20260619 13:14:56.701048 10503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41125 every 8 connection(s)
I20260619 13:14:56.702234 10377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:14:56.704111 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 10377
I20260619 13:14:56.704267 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:14:56.704716 388 external_mini_cluster.cc:1428] Setting key b941835290e74a7dd96cce3ef0e0aecd
I20260619 13:14:56.707245 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38125
--builtin_ntp_servers=127.0.97.20:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 13:14:56.710034 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:56.704617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55293 (local address 127.0.97.62:38125)
0619 13:14:56.704814 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:56.704820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:56.706329 (+ 1509us) server_negotiation.cc:407] Connection header received
0619 13:14:56.707931 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:56.707936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:56.707996 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:56.708083 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:56.709350 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:56.709355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:56.709358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:56.709373 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:56.709402 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:56.709413 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:56.709507 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:56.709689 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 13:14:56.709750 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:14:56.710875 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:56.705152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38125 (local address 127.0.97.2:55293)
0619 13:14:56.706166 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:56.706208 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:56.707692 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:56.708301 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:56.708309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:56.708398 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:56.709069 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:14:56.709071 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:56.709108 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:14:56.709190 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:56.709496 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:56.709505 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:56.709513 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 13:14:56.709755 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 13:14:56.710016 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":308,"threads_started":1}
I20260619 13:14:56.712028 10504 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38125
I20260619 13:14:56.712327 10504 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:56.712936 10504 heartbeater.cc:511] Master 127.0.97.62:38125 requested a full tablet report, sending...
I20260619 13:14:56.714283 10188 ts_manager.cc:194] Registered new tserver with Master: ec1c33eaa72e4574b4b246d12de1fe92 (127.0.97.2:41125)
W20260619 13:14:57.023006 10507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:57.023277 10507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:57.023365 10507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:57.030400 10507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:57.030494 10507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:57.030606 10507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:14:57.039892 10507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:57.041026 10507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:57.042734 10507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:57.054450 10512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:57.054785 10515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:57.054829 10513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:57.055220 10507 server_base.cc:1034] running on GCE node
I20260619 13:14:57.055807 10507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:57.056649 10507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:57.057857 10507 hybrid_clock.cc:648] HybridClock initialized: now 1781874897057789 us; error 50 us; skew 500 ppm
I20260619 13:14:57.058151 10507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:14:57.060222 10507 webserver.cc:460] Webserver started at http://127.0.97.3:36589/ using document root <none> and password file <none>
I20260619 13:14:57.060884 10507 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:57.060981 10507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:57.061282 10507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:57.064082 10507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "37b28d4556dc4935a9092eee84c79a65"
format_stamp: "Formatted at 2026-06-19 13:14:57 on dist-test-slave-mkt3"
server_key: "7e7d8b8e103ba7380ed945550c1fb54f"
server_key_iv: "404183fe9b67abc76bce12bea5a4feba"
server_key_version: "encryptionkey@0"
I20260619 13:14:57.064821 10507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "37b28d4556dc4935a9092eee84c79a65"
format_stamp: "Formatted at 2026-06-19 13:14:57 on dist-test-slave-mkt3"
server_key: "7e7d8b8e103ba7380ed945550c1fb54f"
server_key_iv: "404183fe9b67abc76bce12bea5a4feba"
server_key_version: "encryptionkey@0"
I20260619 13:14:57.069831 10507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:14:57.073127 10521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:57.074390 10507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:14:57.074529 10507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "37b28d4556dc4935a9092eee84c79a65"
format_stamp: "Formatted at 2026-06-19 13:14:57 on dist-test-slave-mkt3"
server_key: "7e7d8b8e103ba7380ed945550c1fb54f"
server_key_iv: "404183fe9b67abc76bce12bea5a4feba"
server_key_version: "encryptionkey@0"
I20260619 13:14:57.074726 10507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:57.080992 10507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:57.081686 10507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:57.081971 10507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:57.082788 10507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:57.083992 10507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:57.084071 10507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:57.084146 10507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:57.084195 10507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:57.111936 10507 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40131
I20260619 13:14:57.111980 10633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40131 every 8 connection(s)
I20260619 13:14:57.113133 10507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:14:57.117523 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 10507
I20260619 13:14:57.117705 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:14:57.118323 388 external_mini_cluster.cc:1428] Setting key 5457a1a43a118d1224f36f7f26359f65
I20260619 13:14:57.121316 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:57.115458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43545 (local address 127.0.97.62:38125)
0619 13:14:57.115633 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:57.115638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:57.116911 (+ 1273us) server_negotiation.cc:407] Connection header received
0619 13:14:57.119065 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:57.119069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:57.119117 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:57.119194 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:57.120657 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:57.120662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:57.120667 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:57.120685 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:57.120723 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:57.120737 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:57.120853 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:57.121008 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 13:14:57.121074 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:14:57.122289 10636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:57.115943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38125 (local address 127.0.97.3:43545)
0619 13:14:57.116769 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:57.116805 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:57.118790 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:57.119427 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:57.119438 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:57.119562 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:57.120307 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:14:57.120311 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:57.120357 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:14:57.120481 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:57.120821 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:57.120826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:57.120831 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 13:14:57.121083 (+ 252us) client_negotiation.cc:240] Negotiation successful
0619 13:14:57.121350 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":227,"threads_started":1}
I20260619 13:14:57.123381 10634 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38125
I20260619 13:14:57.123646 10634 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:57.124172 10634 heartbeater.cc:511] Master 127.0.97.62:38125 requested a full tablet report, sending...
I20260619 13:14:57.125372 10188 ts_manager.cc:194] Registered new tserver with Master: 37b28d4556dc4935a9092eee84c79a65 (127.0.97.3:40131)
I20260619 13:14:57.133169 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 13:14:57.139070 10226 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:14:57.137318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54638 (local address 127.0.97.62:38125)
0619 13:14:57.137500 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:57.137507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:57.137921 (+ 414us) server_negotiation.cc:407] Connection header received
0619 13:14:57.138064 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:57.138069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:57.138128 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:57.138222 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:57.138812 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54638: BlockingRecv error: recv got EOF from 127.0.0.1:54638 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:14:57.139384 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 10247
I20260619 13:14:57.142575 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:57.140177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47334 (local address 127.0.97.1:41505)
0619 13:14:57.140640 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:57.140647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:57.140671 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:14:57.140721 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:57.140727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:57.140924 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:57.141051 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:57.141885 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:57.141895 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:57.141910 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:57.141930 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:57.141976 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:57.141990 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:57.142104 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:57.142209 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 13:14:57.142314 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":232,"threads_started":1}
I20260619 13:14:57.143762 10369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:57.221647 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 10247
I20260619 13:14:57.234403 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 10377
I20260619 13:14:57.237733 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:57.235185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.0.97.2:41125)
0619 13:14:57.235656 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:57.235663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:57.235689 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:14:57.235754 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:57.235761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:57.235970 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:57.236112 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:57.237036 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:57.237042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:57.237054 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:57.237071 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:57.237112 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:57.237125 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:57.237237 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:57.237374 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 13:14:57.237472 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":218,"threads_started":1}
I20260619 13:14:57.239013 10499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:57.316357 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 10377
I20260619 13:14:57.328732 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 10507
I20260619 13:14:57.331995 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:57.329558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.0.97.3:40131)
0619 13:14:57.330119 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:57.330127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:57.330156 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:14:57.330221 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:57.330227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:57.330429 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:57.330550 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:57.331391 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:57.331397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:57.331409 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:14:57.331420 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:57.331461 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:14:57.331474 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:57.331623 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:57.331674 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 13:14:57.331756 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":355,"threads_started":1}
I20260619 13:14:57.333192 10629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:57.411602 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 10507
I20260619 13:14:57.423718 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 10157
I20260619 13:14:57.424808 10218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:14:57.521872 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 10157
2026-06-19T13:14:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2161 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:14:57 dist-test-slave-mkt3 krb5kdc[10652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:14:57 dist-test-slave-mkt3 krb5kdc[10652](info): set up 2 sockets
Jun 19 13:14:57 dist-test-slave-mkt3 krb5kdc[10652](info): commencing operation
krb5kdc: starting...
W20260619 13:14:58.526453 388 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:14:58 dist-test-slave-mkt3 krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:14:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:14:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:14:58.726995 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39875
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:59.037014 10665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:59.037293 10665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:59.037386 10665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:59.044350 10665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:14:59.044448 10665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:59.044499 10665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:59.044553 10665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:14:59.044602 10665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:14:59.054016 10665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:59.055315 10665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:59.056555 10665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:59.067672 10670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:59.067711 10671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:59.067735 10673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:59.068626 10665 server_base.cc:1034] running on GCE node
I20260619 13:14:59.069427 10665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:59.070860 10665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:59.072103 10665 hybrid_clock.cc:648] HybridClock initialized: now 1781874899072055 us; error 33 us; skew 500 ppm
I20260619 13:14:59.072435 10665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:59 dist-test-slave-mkt3 krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:59.075428 10665 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:14:59.076831 10665 webserver.cc:460] Webserver started at http://127.0.97.62:33303/ using document root <none> and password file <none>
I20260619 13:14:59.077490 10665 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:59.077605 10665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:59.077914 10665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:59.080760 10665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "66a2bedce1d842369f6fd95e95e63417"
format_stamp: "Formatted at 2026-06-19 13:14:59 on dist-test-slave-mkt3"
server_key: "d8218888ee5c44d14d0c3da853a8257f"
server_key_iv: "0c3621ec3b84c7b5a691b0c038d6c088"
server_key_version: "encryptionkey@0"
I20260619 13:14:59.081563 10665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "66a2bedce1d842369f6fd95e95e63417"
format_stamp: "Formatted at 2026-06-19 13:14:59 on dist-test-slave-mkt3"
server_key: "d8218888ee5c44d14d0c3da853a8257f"
server_key_iv: "0c3621ec3b84c7b5a691b0c038d6c088"
server_key_version: "encryptionkey@0"
I20260619 13:14:59.087414 10665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:14:59.090708 10680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:59.092001 10665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:14:59.092149 10665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "66a2bedce1d842369f6fd95e95e63417"
format_stamp: "Formatted at 2026-06-19 13:14:59 on dist-test-slave-mkt3"
server_key: "d8218888ee5c44d14d0c3da853a8257f"
server_key_iv: "0c3621ec3b84c7b5a691b0c038d6c088"
server_key_version: "encryptionkey@0"
I20260619 13:14:59.092324 10665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:59.105827 10665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:59.109406 10665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:59.109649 10665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:59.125770 10665 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39875
I20260619 13:14:59.125809 10731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39875 every 8 connection(s)
I20260619 13:14:59.127068 10665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:14:59.130452 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:14:59.136329 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 10665
I20260619 13:14:59.136513 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:14:59.137101 388 external_mini_cluster.cc:1428] Setting key f20ba2a2c4766efb6726178279820f55
I20260619 13:14:59.140084 10732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417: Bootstrap starting.
I20260619 13:14:59.143280 10732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417: Neither blocks nor log segments found. Creating new log.
I20260619 13:14:59.144341 10732 log.cc:824] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417: Log is configured to *not* fsync() on all Append() calls
I20260619 13:14:59.147259 10732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417: No bootstrap required, opened a new log
Jun 19 13:14:59 dist-test-slave-mkt3 krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:59.155591 10732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a2bedce1d842369f6fd95e95e63417" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39875 } }
I20260619 13:14:59.155954 10732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:14:59.156059 10732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a2bedce1d842369f6fd95e95e63417, State: Initialized, Role: FOLLOWER
I20260619 13:14:59.156636 10732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a2bedce1d842369f6fd95e95e63417" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39875 } }
I20260619 13:14:59.156848 10732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:14:59.156992 10732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:14:59.157155 10732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:14:59.158813 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:59.138917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.0.97.62:39875)
0619 13:14:59.139622 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:59.139634 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:59.139671 (+ 37us) server_negotiation.cc:407] Connection header received
0619 13:14:59.141287 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:59.141311 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:59.141677 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:59.142044 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:59.143743 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:59.145204 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:59.146894 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:59.147566 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:59.147695 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:59.152034 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:59.152041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:59.152055 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:59.152098 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:59.154957 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:59.155700 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:59.155705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:59.155712 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:59.155864 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:59.156223 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:59.156227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:59.156229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:59.156750 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:59.157018 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:59.157404 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 13:14:59.157780 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":319,"threads_started":1}
I20260619 13:14:59.159528 10732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a2bedce1d842369f6fd95e95e63417" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39875 } }
I20260619 13:14:59.159976 10732 leader_election.cc:302] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66a2bedce1d842369f6fd95e95e63417; no voters:
I20260619 13:14:59.160601 10732 leader_election.cc:288] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:14:59.160692 10737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:14:59.161878 10737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [term 1 LEADER]: Becoming Leader. State: Replica: 66a2bedce1d842369f6fd95e95e63417, State: Running, Role: LEADER
I20260619 13:14:59.162320 10737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a2bedce1d842369f6fd95e95e63417" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39875 } }
I20260619 13:14:59.162566 10732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:14:59.164871 10738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66a2bedce1d842369f6fd95e95e63417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a2bedce1d842369f6fd95e95e63417" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39875 } } }
I20260619 13:14:59.164816 10739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66a2bedce1d842369f6fd95e95e63417. Latest consensus state: current_term: 1 leader_uuid: "66a2bedce1d842369f6fd95e95e63417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a2bedce1d842369f6fd95e95e63417" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39875 } } }
I20260619 13:14:59.165148 10738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:59.165162 10739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417 [sys.catalog]: This master's current role is: LEADER
I20260619 13:14:59.166642 10743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:14:59.171684 10743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:14:59.180197 10743 catalog_manager.cc:1347] Generated new cluster ID: e3dce88210a0466c8cfe98bedebb06a0
I20260619 13:14:59.180294 10743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:14:59.197506 10743 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:14:59.198763 10743 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:14:59.205359 10743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417: Generated new TSK 0
I20260619 13:14:59.206111 10743 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:14:59.263598 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39875
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:14:59.592841 10758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:14:59.593082 10758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:14:59.593171 10758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:14:59.600226 10758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:14:59.600322 10758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:14:59.600436 10758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:14:59.609501 10758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:14:59.610652 10758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:14:59.611887 10758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:14:59.623802 10763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:59.623838 10764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:14:59.623768 10766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:14:59.624763 10758 server_base.cc:1034] running on GCE node
I20260619 13:14:59.625254 10758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:14:59.626068 10758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:14:59.627252 10758 hybrid_clock.cc:648] HybridClock initialized: now 1781874899627208 us; error 29 us; skew 500 ppm
I20260619 13:14:59.627506 10758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:14:59 dist-test-slave-mkt3 krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:14:59.630230 10758 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:14:59.631464 10758 webserver.cc:460] Webserver started at http://127.0.97.1:43829/ using document root <none> and password file <none>
I20260619 13:14:59.632064 10758 fs_manager.cc:359] Metadata directory not provided
I20260619 13:14:59.632161 10758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:14:59.632428 10758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:14:59.635221 10758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "57868a21d8e0498a824d04da714fbd52"
format_stamp: "Formatted at 2026-06-19 13:14:59 on dist-test-slave-mkt3"
server_key: "9e3904ffa3da198cd0498f4020ea72c6"
server_key_iv: "94447b73f20cf7c93982f0f222f19518"
server_key_version: "encryptionkey@0"
I20260619 13:14:59.635885 10758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "57868a21d8e0498a824d04da714fbd52"
format_stamp: "Formatted at 2026-06-19 13:14:59 on dist-test-slave-mkt3"
server_key: "9e3904ffa3da198cd0498f4020ea72c6"
server_key_iv: "94447b73f20cf7c93982f0f222f19518"
server_key_version: "encryptionkey@0"
I20260619 13:14:59.640522 10758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:14:59.643647 10773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:59.644824 10758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 13:14:59.644948 10758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "57868a21d8e0498a824d04da714fbd52"
format_stamp: "Formatted at 2026-06-19 13:14:59 on dist-test-slave-mkt3"
server_key: "9e3904ffa3da198cd0498f4020ea72c6"
server_key_iv: "94447b73f20cf7c93982f0f222f19518"
server_key_version: "encryptionkey@0"
I20260619 13:14:59.645108 10758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:14:59.665401 10758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:14:59.669085 10758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:14:59.669333 10758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:14:59.670171 10758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:14:59.671350 10758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:14:59.671419 10758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:59.671533 10758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:14:59.671581 10758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:14:59.699018 10758 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41679
I20260619 13:14:59.699086 10885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41679 every 8 connection(s)
I20260619 13:14:59.700188 10758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:14:59.705513 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 10758
I20260619 13:14:59.705709 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:14:59.706475 388 external_mini_cluster.cc:1428] Setting key b4132ed589f033a6fa63a56a0ac058ec
Jun 19 13:14:59 dist-test-slave-mkt3 krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:14:59.719936 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:59.702467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38067 (local address 127.0.97.62:39875)
0619 13:14:59.702851 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 13:14:59.702856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:14:59.703713 (+ 857us) server_negotiation.cc:407] Connection header received
0619 13:14:59.705610 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:14:59.705615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:14:59.705703 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:14:59.705805 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:14:59.707698 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:59.708521 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:59.710707 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:14:59.711069 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:14:59.711153 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:59.714653 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:14:59.714658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:14:59.714663 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:14:59.714698 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:14:59.717187 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:59.717871 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:59.717877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:59.717881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:59.717999 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:14:59.718373 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:14:59.718378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:14:59.718380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:14:59.718621 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:14:59.718747 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 13:14:59.719398 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 13:14:59.719597 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":200,"threads_started":1}
I20260619 13:14:59.720832 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:14:59.702912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39875 (local address 127.0.97.1:38067)
0619 13:14:59.703563 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 13:14:59.703604 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:14:59.705290 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:14:59.706099 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:14:59.706112 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:14:59.706845 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:14:59.707543 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:59.707563 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:59.708677 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:59.708683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:59.710564 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:14:59.710579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:14:59.711261 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:14:59.711267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:14:59.711519 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:14:59.712173 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:14:59.712203 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:14:59.714453 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:14:59.717312 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:59.717319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:59.717331 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:59.717721 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:59.718097 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:14:59.718101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:14:59.718103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:14:59.718264 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:14:59.718738 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:14:59.718745 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:14:59.719036 (+ 291us) client_negotiation.cc:769] Sending connection context
0619 13:14:59.719269 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 13:14:59.719753 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":245,"threads_started":1}
I20260619 13:14:59.722270 10886 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39875
I20260619 13:14:59.722522 10886 heartbeater.cc:464] Registering TS with master...
I20260619 13:14:59.723152 10886 heartbeater.cc:511] Master 127.0.97.62:39875 requested a full tablet report, sending...
I20260619 13:14:59.725272 10697 ts_manager.cc:194] Registered new tserver with Master: 57868a21d8e0498a824d04da714fbd52 (127.0.97.1:41679)
I20260619 13:14:59.727324 10697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38067
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:14:59.752017 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39875
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:15:00.053867 10892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:15:00.054126 10892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:15:00.054255 10892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:15:00.061113 10892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:15:00.061211 10892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:15:00.061344 10892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:15:00.070541 10892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:15:00.071705 10892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:15:00.072933 10892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:15:00.084853 10898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:15:00.085429 10900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:15:00.086006 10897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:15:00.086426 10892 server_base.cc:1034] running on GCE node
I20260619 13:15:00.087050 10892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:15:00.088011 10892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:15:00.089205 10892 hybrid_clock.cc:648] HybridClock initialized: now 1781874900089150 us; error 44 us; skew 500 ppm
I20260619 13:15:00.089506 10892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:15:00 dist-test-slave-mkt3 krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:15:00.092350 10892 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:15:00.093681 10892 webserver.cc:460] Webserver started at http://127.0.97.2:34345/ using document root <none> and password file <none>
I20260619 13:15:00.094357 10892 fs_manager.cc:359] Metadata directory not provided
I20260619 13:15:00.094466 10892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:15:00.094727 10892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:15:00.097536 10892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "ea1caa0a09994c3c9dff1c05793cf34f"
format_stamp: "Formatted at 2026-06-19 13:15:00 on dist-test-slave-mkt3"
server_key: "a100bbfab24fba058e64fdd9779d8bc9"
server_key_iv: "2e78a5be596565860e98af26546d042d"
server_key_version: "encryptionkey@0"
I20260619 13:15:00.098251 10892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ea1caa0a09994c3c9dff1c05793cf34f"
format_stamp: "Formatted at 2026-06-19 13:15:00 on dist-test-slave-mkt3"
server_key: "a100bbfab24fba058e64fdd9779d8bc9"
server_key_iv: "2e78a5be596565860e98af26546d042d"
server_key_version: "encryptionkey@0"
I20260619 13:15:00.102898 10892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:15:00.106829 10907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:00.108382 10892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:15:00.108541 10892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "ea1caa0a09994c3c9dff1c05793cf34f"
format_stamp: "Formatted at 2026-06-19 13:15:00 on dist-test-slave-mkt3"
server_key: "a100bbfab24fba058e64fdd9779d8bc9"
server_key_iv: "2e78a5be596565860e98af26546d042d"
server_key_version: "encryptionkey@0"
I20260619 13:15:00.108758 10892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:15:00.128067 10892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:15:00.132014 10892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:15:00.132256 10892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:15:00.133193 10892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:15:00.134491 10892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:15:00.134562 10892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:00.134673 10892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:15:00.134720 10892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:00.163745 10892 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41533
I20260619 13:15:00.163825 11019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41533 every 8 connection(s)
I20260619 13:15:00.164963 10892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:15:00.169762 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 10892
I20260619 13:15:00.169987 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:15:00.170540 388 external_mini_cluster.cc:1428] Setting key 8b2a91d09865902fa44ed7f35db7a1e3
Jun 19 13:15:00 dist-test-slave-mkt3 krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:15:00.185824 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.167425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60729 (local address 127.0.97.62:39875)
0619 13:15:00.167586 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:00.167590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:00.168875 (+ 1285us) server_negotiation.cc:407] Connection header received
0619 13:15:00.171203 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:00.171209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:00.171295 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:00.171380 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:15:00.173085 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.173789 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.175736 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.176128 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.176217 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.179909 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:15:00.179916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:15:00.179921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:15:00.179959 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:15:00.183030 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:00.183699 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:00.183704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:00.183707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:00.183862 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:00.184214 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:00.184218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:00.184221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:00.184513 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:15:00.184638 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:00.185288 (+ 650us) server_negotiation.cc:299] Negotiation successful
0619 13:15:00.185493 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:15:00.186713 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.167846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39875 (local address 127.0.97.2:60729)
0619 13:15:00.168698 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 13:15:00.168756 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 13:15:00.170910 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:15:00.171625 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:15:00.171633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:15:00.172172 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:15:00.172914 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:00.172938 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.173944 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.173948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:00.175619 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:00.175632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.176322 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.176328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:00.176582 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.177206 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:15:00.177235 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:15:00.179689 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:15:00.183164 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:15:00.183171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:15:00.183184 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:15:00.183600 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:15:00.183953 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:15:00.183956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:15:00.183958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:15:00.184125 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:15:00.184646 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:15:00.184652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:15:00.184967 (+ 315us) client_negotiation.cc:769] Sending connection context
0619 13:15:00.185194 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 13:15:00.185569 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":310,"threads_started":1}
I20260619 13:15:00.188066 11020 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39875
I20260619 13:15:00.188349 11020 heartbeater.cc:464] Registering TS with master...
I20260619 13:15:00.189004 11020 heartbeater.cc:511] Master 127.0.97.62:39875 requested a full tablet report, sending...
I20260619 13:15:00.190380 10697 ts_manager.cc:194] Registered new tserver with Master: ea1caa0a09994c3c9dff1c05793cf34f (127.0.97.2:41533)
I20260619 13:15:00.191136 10697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60729
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:15:00.216907 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39875
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:15:00.521518 11025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:15:00.521775 11025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:15:00.521904 11025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:15:00.528941 11025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:15:00.529037 11025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:15:00.529152 11025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:15:00.538451 11025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39875
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:15:00.539593 11025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:15:00.540810 11025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:15:00.552706 11030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:15:00.552732 11033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:15:00.552771 11031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:15:00.553449 11025 server_base.cc:1034] running on GCE node
I20260619 13:15:00.554039 11025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:15:00.554899 11025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:15:00.556080 11025 hybrid_clock.cc:648] HybridClock initialized: now 1781874900556035 us; error 39 us; skew 500 ppm
I20260619 13:15:00.556365 11025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:15:00 dist-test-slave-mkt3 krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:15:00.559226 11025 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:15:00.560580 11025 webserver.cc:460] Webserver started at http://127.0.97.3:44553/ using document root <none> and password file <none>
I20260619 13:15:00.561223 11025 fs_manager.cc:359] Metadata directory not provided
I20260619 13:15:00.561342 11025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:15:00.561607 11025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:15:00.564396 11025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "fbdc4ac6991a465f95b0afcbc5316016"
format_stamp: "Formatted at 2026-06-19 13:15:00 on dist-test-slave-mkt3"
server_key: "d3d9d820d138666b4771907225e46466"
server_key_iv: "6b07fcd07d564bfb90f72e1b03edec9d"
server_key_version: "encryptionkey@0"
I20260619 13:15:00.565120 11025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fbdc4ac6991a465f95b0afcbc5316016"
format_stamp: "Formatted at 2026-06-19 13:15:00 on dist-test-slave-mkt3"
server_key: "d3d9d820d138666b4771907225e46466"
server_key_iv: "6b07fcd07d564bfb90f72e1b03edec9d"
server_key_version: "encryptionkey@0"
I20260619 13:15:00.569770 11025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 13:15:00.572840 11040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:00.574031 11025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260619 13:15:00.574162 11025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "fbdc4ac6991a465f95b0afcbc5316016"
format_stamp: "Formatted at 2026-06-19 13:15:00 on dist-test-slave-mkt3"
server_key: "d3d9d820d138666b4771907225e46466"
server_key_iv: "6b07fcd07d564bfb90f72e1b03edec9d"
server_key_version: "encryptionkey@0"
I20260619 13:15:00.574330 11025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:15:00.591502 11025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:15:00.595091 11025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:15:00.595352 11025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:15:00.596248 11025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:15:00.597483 11025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:15:00.597556 11025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:00.597668 11025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:15:00.597718 11025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:00.625883 11025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45347
I20260619 13:15:00.625936 11152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45347 every 8 connection(s)
I20260619 13:15:00.627045 11025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:15:00.635898 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 11025
I20260619 13:15:00.636059 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:15:00.636447 388 external_mini_cluster.cc:1428] Setting key f9f3f20afb124c416d5bba580fce4e4c
Jun 19 13:15:00 dist-test-slave-mkt3 krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:15:00.644479 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.629292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38407 (local address 127.0.97.62:39875)
0619 13:15:00.629439 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:00.629443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:00.630568 (+ 1125us) server_negotiation.cc:407] Connection header received
0619 13:15:00.632219 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:00.632223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:00.632293 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:00.632376 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:15:00.633727 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.634471 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.635945 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.636428 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.636541 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.639849 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:15:00.639854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:15:00.639857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:15:00.639888 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:15:00.642136 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:00.642749 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:00.642753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:00.642755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:00.642869 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:00.643181 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:00.643183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:00.643185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:00.643438 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:15:00.643540 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:00.644086 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 13:15:00.644231 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 13:15:00.645313 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.629718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39875 (local address 127.0.97.3:38407)
0619 13:15:00.630412 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 13:15:00.630452 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 13:15:00.631989 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:15:00.632562 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:15:00.632569 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:15:00.633059 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:15:00.633586 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:00.633602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.634586 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.634589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:00.635800 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:00.635812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.636640 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.636646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:00.636874 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.637463 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:15:00.637489 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:15:00.639679 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:15:00.642262 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:15:00.642268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:15:00.642280 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:15:00.642643 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:15:00.642951 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:15:00.642954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:15:00.642956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:15:00.643109 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:15:00.643522 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:15:00.643528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:15:00.643796 (+ 268us) client_negotiation.cc:769] Sending connection context
0619 13:15:00.644002 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 13:15:00.644340 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":253,"threads_started":1}
I20260619 13:15:00.646466 11153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39875
I20260619 13:15:00.646699 11153 heartbeater.cc:464] Registering TS with master...
I20260619 13:15:00.647300 11153 heartbeater.cc:511] Master 127.0.97.62:39875 requested a full tablet report, sending...
I20260619 13:15:00.648497 10697 ts_manager.cc:194] Registered new tserver with Master: fbdc4ac6991a465f95b0afcbc5316016 (127.0.97.3:45347)
I20260619 13:15:00.649295 10697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38407
I20260619 13:15:00.651316 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:15:00.665827 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.655265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.0.97.62:39875)
0619 13:15:00.655435 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:00.655440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:00.655844 (+ 404us) server_negotiation.cc:407] Connection header received
0619 13:15:00.655956 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:00.655959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:00.656012 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:00.656081 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:15:00.657294 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.658017 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.659156 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.659553 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.659639 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.661543 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:15:00.661547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:15:00.661550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:15:00.661575 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:15:00.663759 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:00.664297 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:00.664301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:00.664303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:00.664403 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:00.664669 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:00.664671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:00.664673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:00.664958 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:15:00.665118 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:00.665415 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 13:15:00.665559 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:15:00.669303 10697 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:15:00.675920 10697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:15:00.713228 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.686535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41533 (local address 127.0.0.1:33096)
0619 13:15:00.687390 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0619 13:15:00.687433 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:15:00.687592 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:15:00.706044 (+ 18452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:15:00.706053 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:15:00.706086 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:15:00.706442 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:00.706454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.708657 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.708663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:00.710527 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:00.710544 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.712287 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.712294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:00.712494 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.712547 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 13:15:00.712653 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 13:15:00.712821 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":1,"thread_start_us":20151,"threads_started":1}
I20260619 13:15:00.713973 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.686760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45347 (local address 127.0.0.1:42738)
0619 13:15:00.700629 (+ 13869us) negotiation.cc:107] Waiting for socket to connect
0619 13:15:00.700649 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:15:00.700771 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:15:00.707276 (+ 6505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:15:00.707280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:15:00.707303 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:15:00.707659 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:00.707673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.709577 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.709582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:00.711420 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:00.711433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.713077 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.713084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:00.713247 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.713288 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 13:15:00.713371 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 13:15:00.713625 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13743,"mutex_wait_us":51,"thread_start_us":365,"threads_started":1}
I20260619 13:15:00.714411 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.686671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33096 (local address 127.0.97.2:41533)
0619 13:15:00.705394 (+ 18723us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:00.705402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:00.705436 (+ 34us) server_negotiation.cc:407] Connection header received
0619 13:15:00.705534 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:00.705541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:00.705793 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:00.705994 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:15:00.706631 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.708543 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.710690 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.712080 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.713695 (+ 1615us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.713901 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:00.713973 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 13:15:00.714122 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18552,"thread_start_us":19456,"threads_started":1}
I20260619 13:15:00.714501 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.687954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41679 (local address 127.0.0.1:52132)
0619 13:15:00.706962 (+ 19008us) negotiation.cc:107] Waiting for socket to connect
0619 13:15:00.706978 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:15:00.707093 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:15:00.707614 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:15:00.707618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:15:00.707645 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:15:00.708067 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:00.708079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.709800 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.709806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:00.711826 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:00.711840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.713763 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.713770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:00.713967 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.713998 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:15:00.714083 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 13:15:00.714188 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18887,"mutex_wait_us":1,"thread_start_us":12895,"threads_started":1}
I20260619 13:15:00.714792 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.687305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.0.97.3:45347)
0619 13:15:00.706743 (+ 19438us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:00.706750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:00.706785 (+ 35us) server_negotiation.cc:407] Connection header received
0619 13:15:00.706873 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:00.706880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:00.707108 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:00.707273 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:15:00.707892 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.709448 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.711587 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.712893 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.712995 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.713209 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:00.714376 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0619 13:15:00.714492 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19269,"thread_start_us":19144,"threads_started":1}
I20260619 13:15:00.715215 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.688175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.0.97.1:41679)
0619 13:15:00.703509 (+ 15334us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:00.703518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:00.707076 (+ 3558us) server_negotiation.cc:407] Connection header received
0619 13:15:00.707214 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:00.707221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:00.707428 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:00.707608 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:15:00.708247 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.709670 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.711998 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.713307 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.713418 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.713627 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:00.714798 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0619 13:15:00.714942 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15156,"thread_start_us":15337,"threads_started":1}
I20260619 13:15:00.718207 11088 tablet_service.cc:1467] Processing CreateTablet for tablet 78921bf2349c41478c5a8ef72c3ad761 (DEFAULT_TABLE table=test-table [id=c8a67ba4d2c942d79be1b5d313cd0591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:15:00.718207 10955 tablet_service.cc:1467] Processing CreateTablet for tablet 78921bf2349c41478c5a8ef72c3ad761 (DEFAULT_TABLE table=test-table [id=c8a67ba4d2c942d79be1b5d313cd0591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:15:00.718297 10821 tablet_service.cc:1467] Processing CreateTablet for tablet 78921bf2349c41478c5a8ef72c3ad761 (DEFAULT_TABLE table=test-table [id=c8a67ba4d2c942d79be1b5d313cd0591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:15:00.719489 10821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78921bf2349c41478c5a8ef72c3ad761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:15:00.719547 11088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78921bf2349c41478c5a8ef72c3ad761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:15:00.719548 10955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78921bf2349c41478c5a8ef72c3ad761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:15:00.730422 10886 heartbeater.cc:503] Master 127.0.97.62:39875 was elected leader, sending a full tablet report...
I20260619 13:15:00.734388 11171 tablet_bootstrap.cc:492] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52: Bootstrap starting.
I20260619 13:15:00.737387 11171 tablet_bootstrap.cc:654] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52: Neither blocks nor log segments found. Creating new log.
I20260619 13:15:00.738212 11172 tablet_bootstrap.cc:492] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f: Bootstrap starting.
I20260619 13:15:00.739003 11173 tablet_bootstrap.cc:492] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016: Bootstrap starting.
I20260619 13:15:00.739638 11171 log.cc:824] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52: Log is configured to *not* fsync() on all Append() calls
I20260619 13:15:00.742043 11173 tablet_bootstrap.cc:654] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016: Neither blocks nor log segments found. Creating new log.
I20260619 13:15:00.742175 11172 tablet_bootstrap.cc:654] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f: Neither blocks nor log segments found. Creating new log.
I20260619 13:15:00.742702 11171 tablet_bootstrap.cc:492] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52: No bootstrap required, opened a new log
I20260619 13:15:00.742962 11171 ts_tablet_manager.cc:1397] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260619 13:15:00.743103 11172 log.cc:824] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:15:00.743152 11173 log.cc:824] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016: Log is configured to *not* fsync() on all Append() calls
I20260619 13:15:00.745532 11173 tablet_bootstrap.cc:492] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016: No bootstrap required, opened a new log
I20260619 13:15:00.745755 11172 tablet_bootstrap.cc:492] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f: No bootstrap required, opened a new log
I20260619 13:15:00.745751 11173 ts_tablet_manager.cc:1397] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260619 13:15:00.746026 11172 ts_tablet_manager.cc:1397] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:15:00.752779 11171 raft_consensus.cc:348] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1caa0a09994c3c9dff1c05793cf34f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41533 } } peers { permanent_uuid: "fbdc4ac6991a465f95b0afcbc5316016" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45347 } } peers { permanent_uuid: "57868a21d8e0498a824d04da714fbd52" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41679 } }
I20260619 13:15:00.753181 11171 raft_consensus.cc:374] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:15:00.753293 11171 raft_consensus.cc:729] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57868a21d8e0498a824d04da714fbd52, State: Initialized, Role: FOLLOWER
I20260619 13:15:00.753105 11173 raft_consensus.cc:348] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1caa0a09994c3c9dff1c05793cf34f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41533 } } peers { permanent_uuid: "fbdc4ac6991a465f95b0afcbc5316016" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45347 } } peers { permanent_uuid: "57868a21d8e0498a824d04da714fbd52" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41679 } }
I20260619 13:15:00.753470 11173 raft_consensus.cc:374] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:15:00.753645 11173 raft_consensus.cc:729] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbdc4ac6991a465f95b0afcbc5316016, State: Initialized, Role: FOLLOWER
I20260619 13:15:00.753911 11171 consensus_queue.cc:260] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1caa0a09994c3c9dff1c05793cf34f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41533 } } peers { permanent_uuid: "fbdc4ac6991a465f95b0afcbc5316016" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45347 } } peers { permanent_uuid: "57868a21d8e0498a824d04da714fbd52" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41679 } }
I20260619 13:15:00.754238 11173 consensus_queue.cc:260] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1caa0a09994c3c9dff1c05793cf34f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41533 } } peers { permanent_uuid: "fbdc4ac6991a465f95b0afcbc5316016" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45347 } } peers { permanent_uuid: "57868a21d8e0498a824d04da714fbd52" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41679 } }
I20260619 13:15:00.755550 11172 raft_consensus.cc:348] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1caa0a09994c3c9dff1c05793cf34f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41533 } } peers { permanent_uuid: "fbdc4ac6991a465f95b0afcbc5316016" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45347 } } peers { permanent_uuid: "57868a21d8e0498a824d04da714fbd52" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41679 } }
I20260619 13:15:00.755988 11172 raft_consensus.cc:374] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:15:00.756121 11172 raft_consensus.cc:729] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1caa0a09994c3c9dff1c05793cf34f, State: Initialized, Role: FOLLOWER
I20260619 13:15:00.756559 11171 ts_tablet_manager.cc:1428] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260619 13:15:00.756799 11172 consensus_queue.cc:260] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1caa0a09994c3c9dff1c05793cf34f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41533 } } peers { permanent_uuid: "fbdc4ac6991a465f95b0afcbc5316016" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45347 } } peers { permanent_uuid: "57868a21d8e0498a824d04da714fbd52" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41679 } }
I20260619 13:15:00.757421 11153 heartbeater.cc:503] Master 127.0.97.62:39875 was elected leader, sending a full tablet report...
I20260619 13:15:00.758049 11173 ts_tablet_manager.cc:1428] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260619 13:15:00.760363 11172 ts_tablet_manager.cc:1428] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 13:15:00.760825 11020 heartbeater.cc:503] Master 127.0.97.62:39875 was elected leader, sending a full tablet report...
I20260619 13:15:00.810726 11178 raft_consensus.cc:482] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:15:00.811012 11178 raft_consensus.cc:504] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1caa0a09994c3c9dff1c05793cf34f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41533 } } peers { permanent_uuid: "fbdc4ac6991a465f95b0afcbc5316016" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45347 } } peers { permanent_uuid: "57868a21d8e0498a824d04da714fbd52" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41679 } }
I20260619 13:15:00.812734 11178 leader_election.cc:288] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea1caa0a09994c3c9dff1c05793cf34f (127.0.97.2:41533), 57868a21d8e0498a824d04da714fbd52 (127.0.97.1:41679)
I20260619 13:15:00.817898 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.812984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41679 (local address 127.0.97.3:46887)
0619 13:15:00.813185 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 13:15:00.813205 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:15:00.813333 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:15:00.813681 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:15:00.813685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:15:00.813712 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:15:00.813945 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:00.813954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.814922 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.814928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:00.816153 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:00.816165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.817221 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.817226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:00.817390 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.817413 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:15:00.817489 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 13:15:00.817610 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 13:15:00.817906 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.813151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46887 (local address 127.0.97.1:41679)
0619 13:15:00.813343 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:00.813350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:00.813371 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:15:00.813474 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:00.813479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:00.813553 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:00.813653 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:15:00.814098 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.814794 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.816300 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.817073 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.817192 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.817288 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:00.817530 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 13:15:00.817655 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:15:00.818811 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.813450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41533 (local address 127.0.97.3:46209)
0619 13:15:00.814221 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0619 13:15:00.814235 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:15:00.814340 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:15:00.814609 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:15:00.814612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:15:00.814633 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:15:00.814924 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:00.814933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.815877 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.815881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:00.817113 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:00.817125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.818263 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.818268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:00.818421 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.818454 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:15:00.818522 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 13:15:00.818600 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":491,"threads_started":1}
I20260619 13:15:00.818914 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.813113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46209 (local address 127.0.97.2:41533)
0619 13:15:00.813293 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:00.813299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:00.814328 (+ 1029us) server_negotiation.cc:407] Connection header received
0619 13:15:00.814456 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:00.814460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:00.814515 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:00.814586 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:15:00.815076 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.815739 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.817261 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.818095 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.818205 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.818296 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:00.818566 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 13:15:00.818685 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:15:00.818883 10841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78921bf2349c41478c5a8ef72c3ad761" candidate_uuid: "fbdc4ac6991a465f95b0afcbc5316016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57868a21d8e0498a824d04da714fbd52" is_pre_election: true
I20260619 13:15:00.819418 10841 raft_consensus.cc:2454] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbdc4ac6991a465f95b0afcbc5316016 in term 0.
I20260619 13:15:00.819830 10975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78921bf2349c41478c5a8ef72c3ad761" candidate_uuid: "fbdc4ac6991a465f95b0afcbc5316016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1caa0a09994c3c9dff1c05793cf34f" is_pre_election: true
I20260619 13:15:00.820113 11041 leader_election.cc:302] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57868a21d8e0498a824d04da714fbd52, fbdc4ac6991a465f95b0afcbc5316016; no voters:
I20260619 13:15:00.820338 10975 raft_consensus.cc:2454] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbdc4ac6991a465f95b0afcbc5316016 in term 0.
I20260619 13:15:00.820560 11178 raft_consensus.cc:2781] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:15:00.820691 11178 raft_consensus.cc:482] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:15:00.820786 11178 raft_consensus.cc:3037] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:15:00.822903 11178 raft_consensus.cc:504] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1caa0a09994c3c9dff1c05793cf34f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41533 } } peers { permanent_uuid: "fbdc4ac6991a465f95b0afcbc5316016" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45347 } } peers { permanent_uuid: "57868a21d8e0498a824d04da714fbd52" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41679 } }
I20260619 13:15:00.823407 11178 leader_election.cc:288] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [CANDIDATE]: Term 1 election: Requested vote from peers ea1caa0a09994c3c9dff1c05793cf34f (127.0.97.2:41533), 57868a21d8e0498a824d04da714fbd52 (127.0.97.1:41679)
I20260619 13:15:00.823737 10975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78921bf2349c41478c5a8ef72c3ad761" candidate_uuid: "fbdc4ac6991a465f95b0afcbc5316016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1caa0a09994c3c9dff1c05793cf34f"
I20260619 13:15:00.823855 10841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78921bf2349c41478c5a8ef72c3ad761" candidate_uuid: "fbdc4ac6991a465f95b0afcbc5316016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57868a21d8e0498a824d04da714fbd52"
I20260619 13:15:00.823932 10975 raft_consensus.cc:3037] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:15:00.824039 10841 raft_consensus.cc:3037] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:15:00.826974 10841 raft_consensus.cc:2454] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbdc4ac6991a465f95b0afcbc5316016 in term 1.
I20260619 13:15:00.826974 10975 raft_consensus.cc:2454] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbdc4ac6991a465f95b0afcbc5316016 in term 1.
I20260619 13:15:00.827342 11041 leader_election.cc:302] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57868a21d8e0498a824d04da714fbd52, fbdc4ac6991a465f95b0afcbc5316016; no voters:
I20260619 13:15:00.827584 11178 raft_consensus.cc:2781] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:15:00.827955 11178 raft_consensus.cc:686] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [term 1 LEADER]: Becoming Leader. State: Replica: fbdc4ac6991a465f95b0afcbc5316016, State: Running, Role: LEADER
I20260619 13:15:00.828385 11178 consensus_queue.cc:237] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1caa0a09994c3c9dff1c05793cf34f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41533 } } peers { permanent_uuid: "fbdc4ac6991a465f95b0afcbc5316016" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45347 } } peers { permanent_uuid: "57868a21d8e0498a824d04da714fbd52" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41679 } }
I20260619 13:15:00.832363 10696 catalog_manager.cc:5495] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbdc4ac6991a465f95b0afcbc5316016 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fbdc4ac6991a465f95b0afcbc5316016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1caa0a09994c3c9dff1c05793cf34f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbdc4ac6991a465f95b0afcbc5316016" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57868a21d8e0498a824d04da714fbd52" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41679 } health_report { overall_health: UNKNOWN } } }
I20260619 13:15:00.860481 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.855986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.0.97.3:45347)
0619 13:15:00.856299 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:00.856304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:00.856319 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:15:00.856391 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:00.856395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:00.856467 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:00.856579 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:15:00.856874 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.857557 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.858745 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.859112 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.859200 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.859570 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:15:00.860037 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:15:00.860139 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:00.860178 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 13:15:00.860234 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260619 13:15:00.868000 10975 raft_consensus.cc:1261] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f [term 1 FOLLOWER]: Refusing update from remote peer fbdc4ac6991a465f95b0afcbc5316016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:15:00.868000 10841 raft_consensus.cc:1261] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52 [term 1 FOLLOWER]: Refusing update from remote peer fbdc4ac6991a465f95b0afcbc5316016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:15:00.868798 11181 consensus_queue.cc:1035] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57868a21d8e0498a824d04da714fbd52" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:15:00.869113 11178 consensus_queue.cc:1035] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea1caa0a09994c3c9dff1c05793cf34f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:15:00.879731 11184 mvcc.cc:205] Tried to move back new op lower bound from 7298559593946505216 to 7298559593796624384. Current Snapshot: MvccSnapshot[applied={T|T < 7298559593946505216}]
I20260619 13:15:00.892092 10697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:15:00.892347 10697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:15:00.895494 10697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66a2bedce1d842369f6fd95e95e63417: Sending DeleteTablet for 3 replicas of tablet 78921bf2349c41478c5a8ef72c3ad761
I20260619 13:15:00.896440 10955 tablet_service.cc:1514] Processing DeleteTablet for tablet 78921bf2349c41478c5a8ef72c3ad761 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:15:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33096
I20260619 13:15:00.896682 11088 tablet_service.cc:1514] Processing DeleteTablet for tablet 78921bf2349c41478c5a8ef72c3ad761 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:15:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42738
I20260619 13:15:00.897023 10821 tablet_service.cc:1514] Processing DeleteTablet for tablet 78921bf2349c41478c5a8ef72c3ad761 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:15:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52132
I20260619 13:15:00.897643 11195 tablet_replica.cc:330] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016: stopping tablet replica
I20260619 13:15:00.898118 11195 raft_consensus.cc:2229] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:15:00.898327 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 10758
I20260619 13:15:00.898690 11195 raft_consensus.cc:2258] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:15:00.903599 11195 ts_tablet_manager.cc:1905] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:15:00.910341 11196 tablet_replica.cc:330] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52: stopping tablet replica
I20260619 13:15:00.910883 11196 raft_consensus.cc:2229] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:15:00.911530 11196 raft_consensus.cc:2258] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:15:00.913439 11195 ts_tablet_manager.cc:1918] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:15:00.913952 11195 log.cc:1196] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/78921bf2349c41478c5a8ef72c3ad761
I20260619 13:15:00.914456 11194 tablet_replica.cc:330] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f: stopping tablet replica
I20260619 13:15:00.914846 11194 raft_consensus.cc:2229] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:15:00.915125 11195 ts_tablet_manager.cc:1939] T 78921bf2349c41478c5a8ef72c3ad761 P fbdc4ac6991a465f95b0afcbc5316016: Deleting consensus metadata
I20260619 13:15:00.915791 11196 ts_tablet_manager.cc:1905] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:15:00.916934 10681 catalog_manager.cc:4841] TS fbdc4ac6991a465f95b0afcbc5316016 (127.0.97.3:45347): tablet 78921bf2349c41478c5a8ef72c3ad761 (table test-table [id=c8a67ba4d2c942d79be1b5d313cd0591]) successfully deleted
I20260619 13:15:00.917584 11194 raft_consensus.cc:2258] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:15:00.919757 11194 ts_tablet_manager.cc:1905] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 13:15:00 dist-test-slave-mkt3 krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:15:00.922477 11196 ts_tablet_manager.cc:1918] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:15:00.922708 11196 log.cc:1196] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/78921bf2349c41478c5a8ef72c3ad761
I20260619 13:15:00.923285 11196 ts_tablet_manager.cc:1939] T 78921bf2349c41478c5a8ef72c3ad761 P 57868a21d8e0498a824d04da714fbd52: Deleting consensus metadata
I20260619 13:15:00.925127 10681 catalog_manager.cc:4841] TS 57868a21d8e0498a824d04da714fbd52 (127.0.97.1:41679): tablet 78921bf2349c41478c5a8ef72c3ad761 (table test-table [id=c8a67ba4d2c942d79be1b5d313cd0591]) successfully deleted
I20260619 13:15:00.926103 11194 ts_tablet_manager.cc:1918] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:15:00.926257 11194 log.cc:1196] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/78921bf2349c41478c5a8ef72c3ad761
I20260619 13:15:00.926733 11194 ts_tablet_manager.cc:1939] T 78921bf2349c41478c5a8ef72c3ad761 P ea1caa0a09994c3c9dff1c05793cf34f: Deleting consensus metadata
I20260619 13:15:00.928247 10684 catalog_manager.cc:4841] TS ea1caa0a09994c3c9dff1c05793cf34f (127.0.97.2:41533): tablet 78921bf2349c41478c5a8ef72c3ad761 (table test-table [id=c8a67ba4d2c942d79be1b5d313cd0591]) successfully deleted
I20260619 13:15:00.928391 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:00.899744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.0.97.1:41679)
0619 13:15:00.899915 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:00.899922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:00.908801 (+ 8879us) server_negotiation.cc:407] Connection header received
0619 13:15:00.909218 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:00.909225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:00.909325 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:00.909397 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:15:00.911115 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.912232 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.915080 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:00.915737 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:00.915838 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:00.921054 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:15:00.921073 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:15:00.921090 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:15:00.921133 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:15:00.925053 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:00.926000 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:00.926009 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:00.926012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:00.926162 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:00.926629 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:00.926634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:00.926638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:00.927063 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:15:00.927368 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:00.927661 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 13:15:00.927868 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:15:00.928963 10881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:15:01.070958 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 10758
I20260619 13:15:01.084981 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 10892
Jun 19 13:15:01 dist-test-slave-mkt3 krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:15:01.101075 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:01.085830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33106 (local address 127.0.97.2:41533)
0619 13:15:01.086012 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:01.086018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:01.086038 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:15:01.086177 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:01.086183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:01.086255 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:01.086350 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:15:01.087512 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:01.088362 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:01.090192 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:01.090827 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:01.090945 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:01.095784 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:15:01.095797 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:15:01.095814 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:15:01.095844 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:15:01.098646 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:01.099371 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:01.099377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:01.099380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:01.099491 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:01.099828 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:01.099830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:01.099832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:01.100164 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:15:01.100377 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:01.100649 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 13:15:01.100795 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 13:15:01.101735 11015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:15:01.187271 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 10892
I20260619 13:15:01.207610 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 11025
Jun 19 13:15:01 dist-test-slave-mkt3 krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:15:01.222759 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:01.208405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.0.97.3:45347)
0619 13:15:01.208585 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:01.208595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:01.208614 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:15:01.208774 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:01.208780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:01.208851 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:01.208941 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:15:01.210145 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:01.210961 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:01.212152 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:01.212594 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:01.212703 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:01.217110 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:15:01.217120 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:15:01.217135 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:15:01.217181 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:15:01.220030 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:01.220788 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:01.220795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:01.220797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:01.220952 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:01.221344 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:01.221347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:01.221349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:01.221653 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:15:01.221899 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:01.222205 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 13:15:01.222403 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:15:01.223424 11148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:15:01.338901 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 11025
I20260619 13:15:01.355602 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 10665
I20260619 13:15:01.356616 10727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:15:01.459288 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 10665
2026-06-19T13:15:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3939 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:15:01 dist-test-slave-mkt3 krb5kdc[11204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:15:01 dist-test-slave-mkt3 krb5kdc[11204](info): set up 2 sockets
Jun 19 13:15:01 dist-test-slave-mkt3 krb5kdc[11204](info): commencing operation
krb5kdc: starting...
W20260619 13:15:02.462473 388 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:15:02 dist-test-slave-mkt3 krb5kdc[11204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:15:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:15:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260619 13:15:02.736435 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwhPxxl/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/postgres -l logfile start
2026-06-19 13:15:04.858 UTC [11234] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 13:15:04.858 UTC [11234] LOG: listening on IPv4 address "127.0.97.20", port 36601
2026-06-19 13:15:04.860 UTC [11234] LOG: listening on Unix socket "/tmp/.s.PGSQL.36601"
2026-06-19 13:15:04.879 UTC [11237] LOG: database system was shut down at 2026-06-19 13:15:03 UTC
2026-06-19 13:15:04.882 UTC [11234] LOG: database system is ready to accept connections
I20260619 13:15:05.774099 388 mini_postgres.cc:96] Postgres bound to 36601
2026-06-19 13:15:05.790 UTC [11245] FATAL: database "slave" does not exist
127.0.97.20:36601 - accepting connections
I20260619 13:15:05.790654 388 mini_ranger.cc:161] Starting Ranger...
I20260619 13:15:05.813599 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 13:15:06.212643 388 mini_ranger.cc:87] Created ranger Postgres database
I20260619 13:15:06.212755 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-admin
2026-06-19 13:15:06,474 [I] DB FLAVOR :POSTGRES
2026-06-19 13:15:06,475 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 13:15:06,475 [I] Checking connection..
2026-06-19 13:15:06,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 13:15:06,727 [I] Checking connection passed.
2026-06-19 13:15:06,727 [I] --------- Verifying version history table ---------
2026-06-19 13:15:06,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:15:06,965 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 13:15:06,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:15:07,218 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 13:15:07,218 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 13:15:07,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 13:15:07.443 UTC [11309] WARNING: there is no transaction in progress
2026-06-19 13:15:07,448 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 13:15:07,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:15:07,688 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 13:15:07,688 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 13:15:07,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 13:15:07,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 13:15:08,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mkt3.c.gcp-upstream.internal','N') ;"
2026-06-19 13:15:08,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 13:15:08,671 [I] Table x_portal_user does not exist in database ranger
2026-06-19 13:15:08,671 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 13:15:08,923 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 13:15:08,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 13:15:09,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 13:15:09,406 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 13:15:09,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 13:15:10.278 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:10.294 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:10.308 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:10.321 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:10.335 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:10.353 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:10.402 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:10.408 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:10.421 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:10.430 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:10.439 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:10.451 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:10.460 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:10.465 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:10.470 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:10.476 UTC [11417] WARNING: there is no transaction in progress
2026-06-19 13:15:11,018 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 13:15:11,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 13:15:11,253 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 13:15:11,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 13:15:11,482 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 13:15:11,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 13:15:11,704 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 13:15:11,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 13:15:11,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 13:15:12,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mkt3.c.gcp-upstream.internal';"
2026-06-19 13:15:12,403 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 13:15:12,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 13:15:12,636 [I] DB_PATCHES have already been applied
I20260619 13:15:12.642129 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwhPxxl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 13:15:12.642210 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260619 13:15:12.654861 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:46439
Jun 19, 2026 1:15:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:15:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:15:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 1:15:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 1:15:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 1:15:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 1:15:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 13:15:13.688850 388 mini_ranger.cc:160] Time spent starting Ranger: real 7.898s user 0.001s sys 0.033s
Jun 19, 2026 1:15:33 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 1:15:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 1:15:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 1:15:33 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 1:15:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 1:15:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 1:15:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 1:15:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 1:15:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 1:15:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 1:15:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 1:15:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 1:15:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 1:15:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 1:15:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 1:15:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 1:15:33 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 1:15:34 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 13:15:34.601181 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 13:15:34 dist-test-slave-mkt3 krb5kdc[11204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874934, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 13:15:34.736244 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 13:15:34.753324 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 13:15:35.449908 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 13:15:35.450064 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-kms
I20260619 13:15:35.450189 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:36601
2026-06-19 13:15:35,837 [I] DB FLAVOR :POSTGRES
2026-06-19 13:15:35,837 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 13:15:35,837 [I] Checking connection
2026-06-19 13:15:35,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 13:15:36,069 [I] connection success
2026-06-19 13:15:36,070 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 13:15:36,070 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 13:15:36,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 13:15:36,309 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 13:15:36,309 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 13:15:36,309 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 13:15:36,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:36601/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 13:15:36,549 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 13:15:36.982290 388 mini_ranger_kms.cc:310] Created kms service
I20260619 13:15:37.152114 388 mini_ranger_kms.cc:326] Created kudu user
I20260619 13:15:37.195815 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 13:15:37 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874934, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:15:37.573570 388 mini_ranger_kms.cc:384] Added ranger policy
I20260619 13:15:37.573931 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 13:15:37.573992 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260619 13:15:37.592660 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11597
I20260619 13:15:37.592846 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:36967
Jun 19, 2026 1:15:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:15:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:15:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 1:15:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 1:15:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 1:15:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 13:15:39.622305 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.886s user 0.121s sys 0.285s
I20260619 13:15:39.622563 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 13:15:39.622797 388 mini_ranger_kms.cc:401] 127.0.97.20:36967/kms/v1/keys
Jun 19 13:15:39 dist-test-slave-mkt3 krb5kdc[11204](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781874939, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 13:15:40 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874939, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 13:15:40 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874939, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 13:15:42 dist-test-slave-mkt3 krb5kdc[11204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:15:42.729139 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36377
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36377
--ranger_config_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36967/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:15:43.069068 11648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:15:43.069334 11648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:15:43.069440 11648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:15:43.076706 11648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:15:43.076805 11648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:15:43.076855 11648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:15:43.076926 11648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:15:43.076983 11648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:15:43.086647 11648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:36967/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36377
--ranger_config_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:15:43.087809 11648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:15:43.089032 11648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:15:43.102772 11656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:15:43.102806 11653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:15:43.102914 11648 server_base.cc:1034] running on GCE node
W20260619 13:15:43.102737 11654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:15:43.103992 11648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:15:43.105315 11648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:15:43.106510 11648 hybrid_clock.cc:648] HybridClock initialized: now 1781874943106471 us; error 33 us; skew 500 ppm
I20260619 13:15:43.106813 11648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:15:43 dist-test-slave-mkt3 krb5kdc[11204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:15:43.109712 11648 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:15:43.111100 11648 webserver.cc:460] Webserver started at http://127.0.97.62:42651/ using document root <none> and password file <none>
I20260619 13:15:43.111689 11648 fs_manager.cc:359] Metadata directory not provided
I20260619 13:15:43.111781 11648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:15:43.112047 11648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:15:43 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:15:43.167485 11648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "97fd091a77044556a38b3b3eb02cfaae"
format_stamp: "Formatted at 2026-06-19 13:15:43 on dist-test-slave-mkt3"
server_key: "f1e1eb92075ea879635578d38c92889d"
server_key_iv: "c125a72fe922ddb97f0cad8419332677"
server_key_version: "kuduclusterkey@0"
I20260619 13:15:43.168326 11648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "97fd091a77044556a38b3b3eb02cfaae"
format_stamp: "Formatted at 2026-06-19 13:15:43 on dist-test-slave-mkt3"
server_key: "f1e1eb92075ea879635578d38c92889d"
server_key_iv: "c125a72fe922ddb97f0cad8419332677"
server_key_version: "kuduclusterkey@0"
I20260619 13:15:43.173635 11648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:15:43.192801 11663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:43.194375 11648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:15:43.194535 11648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "97fd091a77044556a38b3b3eb02cfaae"
format_stamp: "Formatted at 2026-06-19 13:15:43 on dist-test-slave-mkt3"
server_key: "f1e1eb92075ea879635578d38c92889d"
server_key_iv: "c125a72fe922ddb97f0cad8419332677"
server_key_version: "kuduclusterkey@0"
I20260619 13:15:43.194742 11648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:15:43.218107 11648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:15:43.221890 11648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:15:43.222159 11648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:15:43.242833 11648 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36377
I20260619 13:15:43.242853 11714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36377 every 8 connection(s)
I20260619 13:15:43.244135 11648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:15:43.246861 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 11648
I20260619 13:15:43.247061 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:15:43.247861 388 external_mini_cluster.cc:1428] Setting key dbcbc1b82d748253497f52f9a6b8a2b7
I20260619 13:15:43.247923 11715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:15:43.264003 11715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae: Bootstrap starting.
I20260619 13:15:43.268528 11715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae: Neither blocks nor log segments found. Creating new log.
I20260619 13:15:43.269591 11715 log.cc:824] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae: Log is configured to *not* fsync() on all Append() calls
I20260619 13:15:43.272552 11715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae: No bootstrap required, opened a new log
I20260619 13:15:43.279996 11715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97fd091a77044556a38b3b3eb02cfaae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } }
I20260619 13:15:43.280341 11715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:15:43.280424 11715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97fd091a77044556a38b3b3eb02cfaae, State: Initialized, Role: FOLLOWER
I20260619 13:15:43.281023 11715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97fd091a77044556a38b3b3eb02cfaae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } }
I20260619 13:15:43.281297 11715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:15:43.281424 11715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:15:43.281577 11715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [term 0 FOLLOWER]: Advancing to term 1
Jun 19 13:15:43 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:15:43.284044 11715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97fd091a77044556a38b3b3eb02cfaae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } }
I20260619 13:15:43.284492 11715 leader_election.cc:302] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97fd091a77044556a38b3b3eb02cfaae; no voters:
I20260619 13:15:43.285153 11715 leader_election.cc:288] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:15:43.285281 11720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:15:43.293619 11715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:15:43.295575 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:43.249976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.0.97.62:36377)
0619 13:15:43.264907 (+ 14931us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:43.264925 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:43.264977 (+ 52us) server_negotiation.cc:407] Connection header received
0619 13:15:43.267314 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:43.267343 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:43.267848 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:43.268262 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:15:43.269868 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:43.271495 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:43.273275 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:43.274265 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:43.274437 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:43.284840 (+ 10403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:15:43.284851 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:15:43.284874 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:15:43.284934 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:15:43.288862 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:43.289755 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:43.289764 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:43.289784 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:43.289985 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:43.290421 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:43.290427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:43.290430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:43.291143 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:15:43.292723 (+ 1580us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:43.292992 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 13:15:43.293379 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14666,"thread_start_us":15435,"threads_started":1}
I20260619 13:15:43.298255 11720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [term 1 LEADER]: Becoming Leader. State: Replica: 97fd091a77044556a38b3b3eb02cfaae, State: Running, Role: LEADER
I20260619 13:15:43.298962 11720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97fd091a77044556a38b3b3eb02cfaae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } }
I20260619 13:15:43.301565 11722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97fd091a77044556a38b3b3eb02cfaae. Latest consensus state: current_term: 1 leader_uuid: "97fd091a77044556a38b3b3eb02cfaae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97fd091a77044556a38b3b3eb02cfaae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } }
I20260619 13:15:43.302062 11722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [sys.catalog]: This master's current role is: LEADER
I20260619 13:15:43.301863 11721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97fd091a77044556a38b3b3eb02cfaae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97fd091a77044556a38b3b3eb02cfaae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } }
I20260619 13:15:43.302572 11721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae [sys.catalog]: This master's current role is: LEADER
I20260619 13:15:43.305795 11729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:15:43.311429 11729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:15:43.312090 11715 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 13:15:43.338446 11729 catalog_manager.cc:1347] Generated new cluster ID: dc34fb9a7fbc4002b9bc5980b2c7468c
I20260619 13:15:43.338586 11729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:15:43.368932 11729 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:15:43.370857 11729 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:15:43.380654 11729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae: Generated new TSK 0
I20260619 13:15:43.381803 11729 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:15:44.174243 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36967/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36377
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:15:44 dist-test-slave-mkt3 krb5kdc[11204](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781874944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 13:15:44.539585 11760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:15:44.539862 11760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:15:44.539992 11760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:15:44.547705 11760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:15:44.547820 11760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:15:44.547995 11760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:15:44.558800 11760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:36967/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:15:44.559988 11760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:15:44.561244 11760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:15:44.575682 11770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:15:44.577332 11771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:15:44.583365 11773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:15:44.583966 11760 server_base.cc:1034] running on GCE node
I20260619 13:15:44.584692 11760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:15:44.585877 11760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:15:44.587122 11760 hybrid_clock.cc:648] HybridClock initialized: now 1781874944587066 us; error 44 us; skew 500 ppm
I20260619 13:15:44.587507 11760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:15:44 dist-test-slave-mkt3 krb5kdc[11204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:15:44.591339 11760 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:15:44.593317 11760 webserver.cc:460] Webserver started at http://127.0.97.1:37933/ using document root <none> and password file <none>
I20260619 13:15:44.594259 11760 fs_manager.cc:359] Metadata directory not provided
I20260619 13:15:44.594435 11760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:15:44.594779 11760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:15:44 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:15:44.615244 11760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "05c7c279c6d44b0d8465651ad5cb5774"
format_stamp: "Formatted at 2026-06-19 13:15:44 on dist-test-slave-mkt3"
server_key: "bf1c99fc1f34179993206dab69fad44b"
server_key_iv: "6f686609d12090bb68343fe203754813"
server_key_version: "kuduclusterkey@0"
I20260619 13:15:44.616350 11760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "05c7c279c6d44b0d8465651ad5cb5774"
format_stamp: "Formatted at 2026-06-19 13:15:44 on dist-test-slave-mkt3"
server_key: "bf1c99fc1f34179993206dab69fad44b"
server_key_iv: "6f686609d12090bb68343fe203754813"
server_key_version: "kuduclusterkey@0"
I20260619 13:15:44.624037 11760 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260619 13:15:44.642180 11780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:44.643596 11760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:15:44.643751 11760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "05c7c279c6d44b0d8465651ad5cb5774"
format_stamp: "Formatted at 2026-06-19 13:15:44 on dist-test-slave-mkt3"
server_key: "bf1c99fc1f34179993206dab69fad44b"
server_key_iv: "6f686609d12090bb68343fe203754813"
server_key_version: "kuduclusterkey@0"
I20260619 13:15:44.643955 11760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:15:44.734385 11760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:15:44.739807 11760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:15:44.740229 11760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:15:44.741504 11760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:15:44.743892 11760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:15:44.743983 11760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:44.744150 11760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:15:44.744213 11760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:44.790179 11760 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33201
I20260619 13:15:44.790259 11892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33201 every 8 connection(s)
I20260619 13:15:44.791478 11760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:15:44.794270 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 11760
I20260619 13:15:44.794485 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:15:44.795329 388 external_mini_cluster.cc:1428] Setting key 9536b3d6351e3db3b90a478143d0fe61
Jun 19 13:15:44 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:15:44.826310 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:44.798550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36377 (local address 127.0.97.1:47119)
0619 13:15:44.802844 (+ 4294us) negotiation.cc:107] Waiting for socket to connect
0619 13:15:44.802909 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 13:15:44.805263 (+ 2354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:15:44.806895 (+ 1632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:15:44.806913 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:15:44.807559 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:15:44.808393 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:44.808419 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:44.810036 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:44.810041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:44.812160 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:44.812176 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:44.813445 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:44.813452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:44.813746 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:44.814524 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:15:44.814552 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:15:44.817081 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:15:44.821531 (+ 4450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:15:44.821540 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:15:44.821559 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:15:44.822186 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:15:44.822635 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:15:44.822641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:15:44.822645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:15:44.822883 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:15:44.823577 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:15:44.823585 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:15:44.824033 (+ 448us) client_negotiation.cc:769] Sending connection context
0619 13:15:44.824327 (+ 294us) client_negotiation.cc:240] Negotiation successful
0619 13:15:44.824867 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3965,"thread_start_us":3709,"threads_started":1}
I20260619 13:15:44.827443 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:44.799200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47119 (local address 127.0.97.62:36377)
0619 13:15:44.805982 (+ 6782us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:44.805989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:44.806009 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:15:44.806067 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:44.806073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:44.806149 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:44.806252 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:15:44.808604 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:44.809783 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:44.812333 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:44.812987 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:44.813086 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:44.817313 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:15:44.817320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:15:44.817326 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:15:44.817374 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:15:44.821348 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:44.822334 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:44.822340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:44.822344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:44.822492 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:44.823017 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:44.823023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:44.823027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:44.823366 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:15:44.823496 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:44.826846 (+ 3350us) server_negotiation.cc:299] Negotiation successful
0619 13:15:44.827080 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6659,"thread_start_us":6513,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:15:44.829754 11893 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36377
I20260619 13:15:44.830114 11893 heartbeater.cc:464] Registering TS with master...
I20260619 13:15:44.830987 11893 heartbeater.cc:511] Master 127.0.97.62:36377 requested a full tablet report, sending...
I20260619 13:15:44.833587 11680 ts_manager.cc:194] Registered new tserver with Master: 05c7c279c6d44b0d8465651ad5cb5774 (127.0.97.1:33201)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:15:44.835671 11680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:15:44.860598 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36967/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36377
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:15:44 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 13:15:45.309561 11899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:15:45.310016 11899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:15:45.310176 11899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Jun 19 13:15:45 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 13:15:45.321502 11899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:15:45.321705 11899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:15:45.321947 11899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:15:45.337374 11899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:36967/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:15:45.339309 11899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:15:45.341034 11899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:15:45.374105 11907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:15:45.377629 11905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:15:45.380124 11910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:15:45.380507 11899 server_base.cc:1034] running on GCE node
I20260619 13:15:45.381556 11899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:15:45.390787 11899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:15:45.392258 11899 hybrid_clock.cc:648] HybridClock initialized: now 1781874945392132 us; error 113 us; skew 500 ppm
I20260619 13:15:45.392764 11899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:15:45 dist-test-slave-mkt3 krb5kdc[11204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:15:45.397531 11899 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:15:45.418036 11899 webserver.cc:460] Webserver started at http://127.0.97.2:33953/ using document root <none> and password file <none>
I20260619 13:15:45.419126 11899 fs_manager.cc:359] Metadata directory not provided
I20260619 13:15:45.419349 11899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:15:45.419780 11899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:15:45 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:15:45.441185 11899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "4afeb4d8cbe44b95ba34688123976a43"
format_stamp: "Formatted at 2026-06-19 13:15:45 on dist-test-slave-mkt3"
server_key: "7e8846717a702be18562fa7005f500b6"
server_key_iv: "ff94d6cf35f96c5a13bff24305ac2e54"
server_key_version: "kuduclusterkey@0"
I20260619 13:15:45.442652 11899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4afeb4d8cbe44b95ba34688123976a43"
format_stamp: "Formatted at 2026-06-19 13:15:45 on dist-test-slave-mkt3"
server_key: "7e8846717a702be18562fa7005f500b6"
server_key_iv: "ff94d6cf35f96c5a13bff24305ac2e54"
server_key_version: "kuduclusterkey@0"
I20260619 13:15:45.477437 11899 fs_manager.cc:691] Time spent creating directory manager: real 0.034s user 0.019s sys 0.012s
I20260619 13:15:45.525223 11916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:45.530898 11899 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.012s sys 0.004s
I20260619 13:15:45.531216 11899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "4afeb4d8cbe44b95ba34688123976a43"
format_stamp: "Formatted at 2026-06-19 13:15:45 on dist-test-slave-mkt3"
server_key: "7e8846717a702be18562fa7005f500b6"
server_key_iv: "ff94d6cf35f96c5a13bff24305ac2e54"
server_key_version: "kuduclusterkey@0"
I20260619 13:15:45.531586 11899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:15:45.603837 11899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:15:45.610533 11899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:15:45.610915 11899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:15:45.612968 11899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:15:45.614521 11899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:15:45.614599 11899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:45.614741 11899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:15:45.614794 11899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:45.660295 11899 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33065
I20260619 13:15:45.660356 12035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33065 every 8 connection(s)
I20260619 13:15:45.661548 11899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:15:45.663398 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 11899
I20260619 13:15:45.663552 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:15:45.663973 388 external_mini_cluster.cc:1428] Setting key 54a26c5b505a01cbaf48d05a2fdf2a9c
Jun 19 13:15:45 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:15:45.685794 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:45.664613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34955 (local address 127.0.97.62:36377)
0619 13:15:45.670196 (+ 5583us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:45.670200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:45.670215 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:15:45.670261 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:45.670265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:45.670337 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:45.670430 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:15:45.671878 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:45.672581 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:45.674154 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:45.674704 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:45.674867 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:45.678979 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:15:45.678985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:15:45.678990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:15:45.679036 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:15:45.682390 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:45.683256 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:45.683262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:45.683265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:45.683451 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:45.683967 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:45.683972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:45.683974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:45.684332 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:15:45.684477 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:45.685200 (+ 723us) server_negotiation.cc:299] Negotiation successful
0619 13:15:45.685415 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5486,"thread_start_us":5387,"threads_started":1}
I20260619 13:15:45.686872 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:45.665240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36377 (local address 127.0.97.2:34955)
0619 13:15:45.666086 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0619 13:15:45.666130 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 13:15:45.667747 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:15:45.670696 (+ 2949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:15:45.670704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:15:45.671128 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:15:45.671711 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:45.671730 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:45.672734 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:45.672738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:45.674020 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:45.674033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:45.674971 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:45.674980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:45.675297 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:45.676042 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:15:45.676071 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:15:45.678750 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:15:45.682551 (+ 3801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:15:45.682560 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:15:45.682578 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:15:45.683114 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:15:45.683582 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:15:45.683587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:15:45.683590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:15:45.683837 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:15:45.684480 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:15:45.684486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:15:45.684818 (+ 332us) client_negotiation.cc:769] Sending connection context
0619 13:15:45.685074 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 13:15:45.685641 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":304,"threads_started":1}
I20260619 13:15:45.688251 12036 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36377
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:15:45.688539 12036 heartbeater.cc:464] Registering TS with master...
I20260619 13:15:45.689208 12036 heartbeater.cc:511] Master 127.0.97.62:36377 requested a full tablet report, sending...
I20260619 13:15:45.690650 11680 ts_manager.cc:194] Registered new tserver with Master: 4afeb4d8cbe44b95ba34688123976a43 (127.0.97.2:33065)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 13:15:45.691931 11680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:15:45.720865 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:36967/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36377
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:15:45.839181 11893 heartbeater.cc:503] Master 127.0.97.62:36377 was elected leader, sending a full tablet report...
W20260619 13:15:46.068524 12042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:15:46.068804 12042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:15:46.068933 12042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:15:46.075932 12042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:15:46.076046 12042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:15:46.076190 12042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:15:46.085469 12042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:36967/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:15:46.086748 12042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:15:46.088016 12042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:15:46.100508 12047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:15:46.100479 12048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:15:46.100890 12050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:15:46.101543 12042 server_base.cc:1034] running on GCE node
I20260619 13:15:46.102144 12042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:15:46.103055 12042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:15:46.104264 12042 hybrid_clock.cc:648] HybridClock initialized: now 1781874946104208 us; error 46 us; skew 500 ppm
I20260619 13:15:46.104588 12042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:15:46 dist-test-slave-mkt3 krb5kdc[11204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:15:46.107895 12042 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:15:46.109294 12042 webserver.cc:460] Webserver started at http://127.0.97.3:33991/ using document root <none> and password file <none>
I20260619 13:15:46.109963 12042 fs_manager.cc:359] Metadata directory not provided
I20260619 13:15:46.110064 12042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:15:46.110313 12042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:15:46 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:15:46.126951 12042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1"
format_stamp: "Formatted at 2026-06-19 13:15:46 on dist-test-slave-mkt3"
server_key: "e4502d5a6537fc4c0a3b12bb3cf66fd7"
server_key_iv: "6d68f804a4fc508a86a5f3abdbf673fc"
server_key_version: "kuduclusterkey@0"
I20260619 13:15:46.127727 12042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1"
format_stamp: "Formatted at 2026-06-19 13:15:46 on dist-test-slave-mkt3"
server_key: "e4502d5a6537fc4c0a3b12bb3cf66fd7"
server_key_iv: "6d68f804a4fc508a86a5f3abdbf673fc"
server_key_version: "kuduclusterkey@0"
I20260619 13:15:46.132726 12042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 13:15:46.146384 12057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:46.147823 12042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:15:46.147996 12042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1"
format_stamp: "Formatted at 2026-06-19 13:15:46 on dist-test-slave-mkt3"
server_key: "e4502d5a6537fc4c0a3b12bb3cf66fd7"
server_key_iv: "6d68f804a4fc508a86a5f3abdbf673fc"
server_key_version: "kuduclusterkey@0"
I20260619 13:15:46.148209 12042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:15:46.183477 12042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:15:46.187394 12042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:15:46.187660 12042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:15:46.188536 12042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:15:46.189801 12042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:15:46.189904 12042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:46.190021 12042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:15:46.190073 12042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:15:46.220672 12042 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45939
I20260619 13:15:46.220707 12169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45939 every 8 connection(s)
I20260619 13:15:46.221886 12042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:15:46.223559 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 12042
I20260619 13:15:46.223721 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:15:46.224226 388 external_mini_cluster.cc:1428] Setting key ce7a07704f1dd6662011389116dc45fd
Jun 19 13:15:46 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:15:46.241189 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.224489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37837 (local address 127.0.97.62:36377)
0619 13:15:46.224917 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:46.224923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:46.226245 (+ 1322us) server_negotiation.cc:407] Connection header received
0619 13:15:46.227903 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:46.227907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:46.227969 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:46.228055 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:15:46.229362 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.230160 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.231596 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.232000 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.232090 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.235524 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:15:46.235529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:15:46.235532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:15:46.235561 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:15:46.238308 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:46.239033 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:46.239038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:46.239043 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:46.239178 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:46.239589 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:46.239594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:46.239598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:46.239858 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:15:46.239985 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:46.240622 (+ 637us) server_negotiation.cc:299] Negotiation successful
0619 13:15:46.240826 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":204,"threads_started":1}
I20260619 13:15:46.242260 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.225129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36377 (local address 127.0.97.3:37837)
0619 13:15:46.225936 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0619 13:15:46.225978 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:15:46.227663 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:15:46.228249 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:15:46.228257 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:15:46.228651 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:15:46.229220 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:46.229237 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.230287 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.230291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:46.231479 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:46.231491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.232167 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.232171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:46.232364 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.232939 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:15:46.232960 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:15:46.235317 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:15:46.238428 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:15:46.238437 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:15:46.238453 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:15:46.238934 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:15:46.239275 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:15:46.239280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:15:46.239284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:15:46.239478 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:15:46.239973 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:15:46.239981 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:15:46.240289 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 13:15:46.240528 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 13:15:46.240997 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":289,"threads_started":1}
I20260619 13:15:46.247001 12170 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36377
I20260619 13:15:46.247272 12170 heartbeater.cc:464] Registering TS with master...
I20260619 13:15:46.247960 12170 heartbeater.cc:511] Master 127.0.97.62:36377 requested a full tablet report, sending...
I20260619 13:15:46.249366 11680 ts_manager.cc:194] Registered new tserver with Master: fec0c6415c824271ad3c6ad0f8e4d2b1 (127.0.97.3:45939)
I20260619 13:15:46.250528 11680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37837
I20260619 13:15:46.251302 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:15:46.267894 12172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.255763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37836 (local address 127.0.97.62:36377)
0619 13:15:46.255950 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:46.255958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:46.256373 (+ 415us) server_negotiation.cc:407] Connection header received
0619 13:15:46.256536 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:46.256539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:46.256597 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:46.256674 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:15:46.258063 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.258737 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.260054 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.260469 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.260574 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.262816 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:15:46.262821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:15:46.262824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:15:46.262856 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:15:46.265533 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:46.266166 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:46.266171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:46.266175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:46.266289 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:46.266623 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:46.266636 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:46.266640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:46.266880 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:15:46.267133 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:46.267402 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 13:15:46.267594 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:15:46.272545 11679 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:15:46.279315 11679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:15:46.314532 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.305874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33065 (local address 127.0.0.1:48274)
0619 13:15:46.306377 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 13:15:46.306391 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:15:46.306495 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:15:46.307471 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:15:46.307476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:15:46.307499 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:15:46.307869 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:46.307878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.309420 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.309425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:46.311566 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:46.311580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.313399 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.313405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:46.313584 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.313674 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 13:15:46.313797 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 13:15:46.314085 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":157,"thread_start_us":253,"threads_started":1}
I20260619 13:15:46.314687 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.290939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58576 (local address 127.0.97.1:33201)
0619 13:15:46.305409 (+ 14470us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:46.305416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:46.305441 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:15:46.305519 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:46.305525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:46.305716 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:46.305982 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:15:46.306929 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.308938 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.311909 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.313165 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.313303 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.313539 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:46.314198 (+ 659us) server_negotiation.cc:299] Negotiation successful
0619 13:15:46.314367 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14323,"thread_start_us":14190,"threads_started":1}
I20260619 13:15:46.314725 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.306139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.0.97.2:33065)
0619 13:15:46.306927 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:46.306934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:46.306957 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:15:46.307041 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:46.307047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:46.307268 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:46.308004 (+ 736us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:15:46.308198 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.309289 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.311794 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.313188 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.314031 (+ 843us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.314230 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:46.314287 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 13:15:46.314420 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":539,"threads_started":1}
I20260619 13:15:46.314554 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.291269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45939 (local address 127.0.0.1:54952)
0619 13:15:46.305676 (+ 14407us) negotiation.cc:107] Waiting for socket to connect
0619 13:15:46.305692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:15:46.305812 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:15:46.306929 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:15:46.306933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:15:46.306956 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:15:46.307265 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:46.307277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.309068 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.309073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:46.311717 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:46.311734 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.313773 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.313780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:46.313953 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.313986 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:15:46.314064 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 13:15:46.314168 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14282,"mutex_wait_us":121,"thread_start_us":12846,"threads_started":1}
I20260619 13:15:46.315152 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.291674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54952 (local address 127.0.97.3:45939)
0619 13:15:46.303690 (+ 12016us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:46.303700 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:46.306333 (+ 2633us) server_negotiation.cc:407] Connection header received
0619 13:15:46.306439 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:46.306446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:46.306685 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:46.306925 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:15:46.307471 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.308938 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.311899 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.313573 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.313685 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.313927 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:46.314652 (+ 725us) server_negotiation.cc:299] Negotiation successful
0619 13:15:46.314806 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11798,"thread_start_us":12050,"threads_started":1}
I20260619 13:15:46.315462 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.290674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33201 (local address 127.0.0.1:58576)
0619 13:15:46.304532 (+ 13858us) negotiation.cc:107] Waiting for socket to connect
0619 13:15:46.304587 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 13:15:46.304834 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:15:46.306262 (+ 1428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:15:46.306269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:15:46.306298 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:15:46.306737 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:46.306749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.309482 (+ 2733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.309487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:46.311757 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:46.311767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.313366 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.313373 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:46.313588 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.313674 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 13:15:46.313798 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 13:15:46.314050 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13581,"mutex_wait_us":75,"thread_start_us":14766,"threads_started":1}
I20260619 13:15:46.318915 11971 tablet_service.cc:1467] Processing CreateTablet for tablet dca3fa981a5f4aee9a72cd885257619f (DEFAULT_TABLE table=test-table [id=d5c3819509d24c558f26f71a730af7c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:15:46.318990 12105 tablet_service.cc:1467] Processing CreateTablet for tablet dca3fa981a5f4aee9a72cd885257619f (DEFAULT_TABLE table=test-table [id=d5c3819509d24c558f26f71a730af7c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:15:46.318991 11828 tablet_service.cc:1467] Processing CreateTablet for tablet dca3fa981a5f4aee9a72cd885257619f (DEFAULT_TABLE table=test-table [id=d5c3819509d24c558f26f71a730af7c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:15:46.320039 11971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dca3fa981a5f4aee9a72cd885257619f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:15:46.320322 11828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dca3fa981a5f4aee9a72cd885257619f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:15:46.320493 12105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dca3fa981a5f4aee9a72cd885257619f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:15:46.330291 12189 tablet_bootstrap.cc:492] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43: Bootstrap starting.
I20260619 13:15:46.333905 12190 tablet_bootstrap.cc:492] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1: Bootstrap starting.
I20260619 13:15:46.334805 12189 tablet_bootstrap.cc:654] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43: Neither blocks nor log segments found. Creating new log.
I20260619 13:15:46.335321 12191 tablet_bootstrap.cc:492] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774: Bootstrap starting.
I20260619 13:15:46.336127 12189 log.cc:824] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43: Log is configured to *not* fsync() on all Append() calls
I20260619 13:15:46.337138 12190 tablet_bootstrap.cc:654] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1: Neither blocks nor log segments found. Creating new log.
I20260619 13:15:46.338167 12190 log.cc:824] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1: Log is configured to *not* fsync() on all Append() calls
I20260619 13:15:46.338902 12191 tablet_bootstrap.cc:654] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774: Neither blocks nor log segments found. Creating new log.
I20260619 13:15:46.339565 12189 tablet_bootstrap.cc:492] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43: No bootstrap required, opened a new log
I20260619 13:15:46.339818 12189 ts_tablet_manager.cc:1397] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 13:15:46.339895 12191 log.cc:824] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774: Log is configured to *not* fsync() on all Append() calls
I20260619 13:15:46.340940 12190 tablet_bootstrap.cc:492] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1: No bootstrap required, opened a new log
I20260619 13:15:46.341198 12190 ts_tablet_manager.cc:1397] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260619 13:15:46.342393 12191 tablet_bootstrap.cc:492] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774: No bootstrap required, opened a new log
I20260619 13:15:46.342639 12191 ts_tablet_manager.cc:1397] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:15:46.349128 12191 raft_consensus.cc:348] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c7c279c6d44b0d8465651ad5cb5774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33201 } } peers { permanent_uuid: "4afeb4d8cbe44b95ba34688123976a43" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33065 } } peers { permanent_uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45939 } }
I20260619 13:15:46.349454 12191 raft_consensus.cc:374] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:15:46.349581 12191 raft_consensus.cc:729] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05c7c279c6d44b0d8465651ad5cb5774, State: Initialized, Role: FOLLOWER
I20260619 13:15:46.350198 12191 consensus_queue.cc:260] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c7c279c6d44b0d8465651ad5cb5774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33201 } } peers { permanent_uuid: "4afeb4d8cbe44b95ba34688123976a43" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33065 } } peers { permanent_uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45939 } }
I20260619 13:15:46.351011 12189 raft_consensus.cc:348] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c7c279c6d44b0d8465651ad5cb5774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33201 } } peers { permanent_uuid: "4afeb4d8cbe44b95ba34688123976a43" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33065 } } peers { permanent_uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45939 } }
I20260619 13:15:46.351461 12189 raft_consensus.cc:374] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:15:46.351282 12190 raft_consensus.cc:348] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c7c279c6d44b0d8465651ad5cb5774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33201 } } peers { permanent_uuid: "4afeb4d8cbe44b95ba34688123976a43" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33065 } } peers { permanent_uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45939 } }
I20260619 13:15:46.351621 12189 raft_consensus.cc:729] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4afeb4d8cbe44b95ba34688123976a43, State: Initialized, Role: FOLLOWER
I20260619 13:15:46.351663 12190 raft_consensus.cc:374] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:15:46.351738 12190 raft_consensus.cc:729] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec0c6415c824271ad3c6ad0f8e4d2b1, State: Initialized, Role: FOLLOWER
I20260619 13:15:46.351819 12191 ts_tablet_manager.cc:1428] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260619 13:15:46.352347 12190 consensus_queue.cc:260] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c7c279c6d44b0d8465651ad5cb5774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33201 } } peers { permanent_uuid: "4afeb4d8cbe44b95ba34688123976a43" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33065 } } peers { permanent_uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45939 } }
I20260619 13:15:46.352391 12189 consensus_queue.cc:260] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c7c279c6d44b0d8465651ad5cb5774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33201 } } peers { permanent_uuid: "4afeb4d8cbe44b95ba34688123976a43" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33065 } } peers { permanent_uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45939 } }
I20260619 13:15:46.353396 12170 heartbeater.cc:503] Master 127.0.97.62:36377 was elected leader, sending a full tablet report...
I20260619 13:15:46.354094 12190 ts_tablet_manager.cc:1428] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260619 13:15:46.355755 12036 heartbeater.cc:503] Master 127.0.97.62:36377 was elected leader, sending a full tablet report...
I20260619 13:15:46.356323 12189 ts_tablet_manager.cc:1428] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43: Time spent starting tablet: real 0.016s user 0.007s sys 0.007s
W20260619 13:15:46.413408 12037 tablet.cc:2369] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:15:46.473302 12171 tablet.cc:2369] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:15:46.543962 11894 tablet.cc:2369] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:15:46.627933 12197 raft_consensus.cc:482] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:15:46.628206 12197 raft_consensus.cc:504] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c7c279c6d44b0d8465651ad5cb5774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33201 } } peers { permanent_uuid: "4afeb4d8cbe44b95ba34688123976a43" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33065 } } peers { permanent_uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45939 } }
I20260619 13:15:46.629829 12197 leader_election.cc:288] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05c7c279c6d44b0d8465651ad5cb5774 (127.0.97.1:33201), fec0c6415c824271ad3c6ad0f8e4d2b1 (127.0.97.3:45939)
I20260619 13:15:46.636250 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.630251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53057 (local address 127.0.97.1:33201)
0619 13:15:46.630431 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:46.630436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:46.631415 (+ 979us) server_negotiation.cc:407] Connection header received
0619 13:15:46.631608 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:46.631612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:46.631680 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:46.631755 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:15:46.632258 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.632942 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.634667 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.635422 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.635531 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.635625 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:46.635883 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 13:15:46.636002 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:15:46.636341 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.630743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50673 (local address 127.0.97.3:45939)
0619 13:15:46.630943 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:46.630949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:46.631615 (+ 666us) server_negotiation.cc:407] Connection header received
0619 13:15:46.631774 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:46.631779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:46.631834 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:46.631904 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:15:46.632461 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.633130 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.634753 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.635502 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.635611 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.635703 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:46.635998 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 13:15:46.636114 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 13:15:46.636437 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.630214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33201 (local address 127.0.97.2:53057)
0619 13:15:46.631300 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0619 13:15:46.631315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:15:46.631471 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:15:46.631783 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:15:46.631787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:15:46.631809 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:15:46.632121 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:46.632134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.633059 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.633064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:46.634529 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:46.634540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.635614 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.635620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:46.635744 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.635766 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:15:46.635854 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 13:15:46.635969 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":1,"thread_start_us":690,"threads_started":1}
I20260619 13:15:46.636474 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.630769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45939 (local address 127.0.97.2:50673)
0619 13:15:46.631518 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0619 13:15:46.631532 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:15:46.631653 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:15:46.631942 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:15:46.631946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:15:46.631971 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:15:46.632326 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:46.632338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.633265 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.633270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:46.634607 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:15:46.634619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.635703 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.635709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:15:46.635865 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.635887 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:15:46.635961 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 13:15:46.636069 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":1,"thread_start_us":417,"threads_started":1}
I20260619 13:15:46.637362 12125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca3fa981a5f4aee9a72cd885257619f" candidate_uuid: "4afeb4d8cbe44b95ba34688123976a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1" is_pre_election: true
I20260619 13:15:46.637362 11848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca3fa981a5f4aee9a72cd885257619f" candidate_uuid: "4afeb4d8cbe44b95ba34688123976a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05c7c279c6d44b0d8465651ad5cb5774" is_pre_election: true
I20260619 13:15:46.637825 12125 raft_consensus.cc:2454] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4afeb4d8cbe44b95ba34688123976a43 in term 0.
I20260619 13:15:46.638223 11848 raft_consensus.cc:2454] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4afeb4d8cbe44b95ba34688123976a43 in term 0.
I20260619 13:15:46.638746 11917 leader_election.cc:302] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4afeb4d8cbe44b95ba34688123976a43, fec0c6415c824271ad3c6ad0f8e4d2b1; no voters:
I20260619 13:15:46.639089 12197 raft_consensus.cc:2781] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:15:46.639235 12197 raft_consensus.cc:482] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:15:46.639338 12197 raft_consensus.cc:3037] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:15:46.641368 12197 raft_consensus.cc:504] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c7c279c6d44b0d8465651ad5cb5774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33201 } } peers { permanent_uuid: "4afeb4d8cbe44b95ba34688123976a43" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33065 } } peers { permanent_uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45939 } }
I20260619 13:15:46.641942 12197 leader_election.cc:288] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [CANDIDATE]: Term 1 election: Requested vote from peers 05c7c279c6d44b0d8465651ad5cb5774 (127.0.97.1:33201), fec0c6415c824271ad3c6ad0f8e4d2b1 (127.0.97.3:45939)
I20260619 13:15:46.642270 11848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca3fa981a5f4aee9a72cd885257619f" candidate_uuid: "4afeb4d8cbe44b95ba34688123976a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05c7c279c6d44b0d8465651ad5cb5774"
I20260619 13:15:46.642295 12125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dca3fa981a5f4aee9a72cd885257619f" candidate_uuid: "4afeb4d8cbe44b95ba34688123976a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1"
I20260619 13:15:46.642436 11848 raft_consensus.cc:3037] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:15:46.642444 12125 raft_consensus.cc:3037] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:15:46.644402 12125 raft_consensus.cc:2454] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4afeb4d8cbe44b95ba34688123976a43 in term 1.
I20260619 13:15:46.644407 11848 raft_consensus.cc:2454] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4afeb4d8cbe44b95ba34688123976a43 in term 1.
I20260619 13:15:46.644716 11917 leader_election.cc:302] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4afeb4d8cbe44b95ba34688123976a43, fec0c6415c824271ad3c6ad0f8e4d2b1; no voters:
I20260619 13:15:46.644960 12197 raft_consensus.cc:2781] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:15:46.645345 12197 raft_consensus.cc:686] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [term 1 LEADER]: Becoming Leader. State: Replica: 4afeb4d8cbe44b95ba34688123976a43, State: Running, Role: LEADER
I20260619 13:15:46.645778 12197 consensus_queue.cc:237] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c7c279c6d44b0d8465651ad5cb5774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33201 } } peers { permanent_uuid: "4afeb4d8cbe44b95ba34688123976a43" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33065 } } peers { permanent_uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45939 } }
I20260619 13:15:46.650308 11680 catalog_manager.cc:5495] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4afeb4d8cbe44b95ba34688123976a43 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "4afeb4d8cbe44b95ba34688123976a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05c7c279c6d44b0d8465651ad5cb5774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4afeb4d8cbe44b95ba34688123976a43" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45939 } health_report { overall_health: UNKNOWN } } }
I20260619 13:15:46.678746 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.673779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.0.97.2:33065)
0619 13:15:46.673960 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:46.673965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:46.673982 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:15:46.674145 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:46.674148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:46.674224 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:46.674361 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:15:46.674656 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.675346 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.676751 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.677184 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.677284 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.677748 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:15:46.678298 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:15:46.678424 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:46.678486 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 13:15:46.678548 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 13:15:46.686244 11848 raft_consensus.cc:1261] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774 [term 1 FOLLOWER]: Refusing update from remote peer 4afeb4d8cbe44b95ba34688123976a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:15:46.686244 12125 raft_consensus.cc:1261] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1 [term 1 FOLLOWER]: Refusing update from remote peer 4afeb4d8cbe44b95ba34688123976a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:15:46.687114 12200 consensus_queue.cc:1035] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05c7c279c6d44b0d8465651ad5cb5774" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:15:46.687433 12197 consensus_queue.cc:1035] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec0c6415c824271ad3c6ad0f8e4d2b1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:15:46.697690 12203 mvcc.cc:205] Tried to move back new op lower bound from 7298559781618130944 to 7298559781464653824. Current Snapshot: MvccSnapshot[applied={T|T < 7298559781618130944}]
I20260619 13:15:46.699533 12204 mvcc.cc:205] Tried to move back new op lower bound from 7298559781618130944 to 7298559781464653824. Current Snapshot: MvccSnapshot[applied={T|T < 7298559781618130944}]
I20260619 13:15:46.712149 11679 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:15:46.712393 11679 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:15:46.715443 11679 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97fd091a77044556a38b3b3eb02cfaae: Sending DeleteTablet for 3 replicas of tablet dca3fa981a5f4aee9a72cd885257619f
I20260619 13:15:46.716467 11828 tablet_service.cc:1514] Processing DeleteTablet for tablet dca3fa981a5f4aee9a72cd885257619f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:15:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58576
I20260619 13:15:46.716512 11971 tablet_service.cc:1514] Processing DeleteTablet for tablet dca3fa981a5f4aee9a72cd885257619f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:15:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48274
I20260619 13:15:46.716584 12105 tablet_service.cc:1514] Processing DeleteTablet for tablet dca3fa981a5f4aee9a72cd885257619f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:15:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54952
I20260619 13:15:46.717348 12213 tablet_replica.cc:330] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774: stopping tablet replica
I20260619 13:15:46.717718 12213 raft_consensus.cc:2229] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:15:46.718223 12213 raft_consensus.cc:2258] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:15:46.719533 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 11760
I20260619 13:15:46.724390 12213 ts_tablet_manager.cc:1905] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:15:46.728096 12215 tablet_replica.cc:330] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1: stopping tablet replica
I20260619 13:15:46.728615 12215 raft_consensus.cc:2229] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:15:46.729130 12215 raft_consensus.cc:2258] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:15:46.729877 12214 tablet_replica.cc:330] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43: stopping tablet replica
I20260619 13:15:46.731531 12215 ts_tablet_manager.cc:1905] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:15:46.731556 12213 ts_tablet_manager.cc:1918] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:15:46.731683 12213 log.cc:1196] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/dca3fa981a5f4aee9a72cd885257619f
I20260619 13:15:46.732004 12214 raft_consensus.cc:2229] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:15:46.732224 12213 ts_tablet_manager.cc:1939] T dca3fa981a5f4aee9a72cd885257619f P 05c7c279c6d44b0d8465651ad5cb5774: Deleting consensus metadata
I20260619 13:15:46.732785 12214 raft_consensus.cc:2258] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:15:46.733948 11666 catalog_manager.cc:4841] TS 05c7c279c6d44b0d8465651ad5cb5774 (127.0.97.1:33201): tablet dca3fa981a5f4aee9a72cd885257619f (table test-table [id=d5c3819509d24c558f26f71a730af7c2]) successfully deleted
Jun 19 13:15:46 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:15:46.736174 12214 ts_tablet_manager.cc:1905] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:15:46.738998 12215 ts_tablet_manager.cc:1918] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:15:46.739199 12215 log.cc:1196] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/dca3fa981a5f4aee9a72cd885257619f
I20260619 13:15:46.739691 12215 ts_tablet_manager.cc:1939] T dca3fa981a5f4aee9a72cd885257619f P fec0c6415c824271ad3c6ad0f8e4d2b1: Deleting consensus metadata
I20260619 13:15:46.741315 11664 catalog_manager.cc:4841] TS fec0c6415c824271ad3c6ad0f8e4d2b1 (127.0.97.3:45939): tablet dca3fa981a5f4aee9a72cd885257619f (table test-table [id=d5c3819509d24c558f26f71a730af7c2]) successfully deleted
I20260619 13:15:46.743402 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.720872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.0.97.1:33201)
0619 13:15:46.721031 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:46.721036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:46.725299 (+ 4263us) server_negotiation.cc:407] Connection header received
0619 13:15:46.725481 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:46.725486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:46.725568 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:46.725679 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:15:46.727315 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.728432 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.730587 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.731255 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.731349 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.736501 (+ 5152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:15:46.736514 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:15:46.736531 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:15:46.736579 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:15:46.740303 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:46.741101 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:46.741110 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:46.741114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:46.741259 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:46.741762 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:46.741767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:46.741771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:46.742209 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:15:46.742506 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:46.742801 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 13:15:46.743038 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:15:46.743942 12214 ts_tablet_manager.cc:1918] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:15:46.744127 12214 log.cc:1196] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/dca3fa981a5f4aee9a72cd885257619f
I20260619 13:15:46.744175 11888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:15:46.744679 12214 ts_tablet_manager.cc:1939] T dca3fa981a5f4aee9a72cd885257619f P 4afeb4d8cbe44b95ba34688123976a43: Deleting consensus metadata
I20260619 13:15:46.746259 11666 catalog_manager.cc:4841] TS 4afeb4d8cbe44b95ba34688123976a43 (127.0.97.2:33065): tablet dca3fa981a5f4aee9a72cd885257619f (table test-table [id=d5c3819509d24c558f26f71a730af7c2]) successfully deleted
I20260619 13:15:46.897971 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 11760
I20260619 13:15:46.912472 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 11899
Jun 19 13:15:46 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:15:46.929222 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:46.913168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.0.97.2:33065)
0619 13:15:46.913347 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:46.913353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:46.913373 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:15:46.913496 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:46.913502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:46.913599 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:46.913705 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:15:46.914922 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.916249 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.917558 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:46.918230 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:46.918359 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:46.922657 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:15:46.922675 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:15:46.922695 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:15:46.922750 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:15:46.926540 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:46.927200 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:46.927210 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:46.927214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:46.927396 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:46.927711 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:46.927716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:46.927720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:46.928134 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:15:46.928455 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:46.928615 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 13:15:46.928828 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:15:46.929935 12031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:15:47.041349 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 11899
I20260619 13:15:47.055361 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 12042
Jun 19 13:15:47 dist-test-slave-mkt3 krb5kdc[11204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:15:47.068430 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:15:47.056058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54960 (local address 127.0.97.3:45939)
0619 13:15:47.056206 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 13:15:47.056211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:15:47.056267 (+ 56us) server_negotiation.cc:407] Connection header received
0619 13:15:47.056434 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:15:47.056438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:15:47.056499 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:15:47.056569 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:15:47.057755 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:47.058469 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:47.059675 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:15:47.060043 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:15:47.060122 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:15:47.063957 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:15:47.063970 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:15:47.063986 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:15:47.064019 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:15:47.066389 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:47.066952 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:47.066958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:47.066961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:47.067071 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:15:47.067341 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:15:47.067343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:15:47.067345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:15:47.067650 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:15:47.067853 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 13:15:47.068045 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 13:15:47.068183 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:15:47.068989 12165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:15:47.154075 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 12042
I20260619 13:15:47.166469 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 11648
I20260619 13:15:47.167531 11710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:15:47.277285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 11648
I20260619 13:15:47.294454 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 13:15:47.628541 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 13:15:47.628753 388 mini_ranger.cc:66] Stopping Ranger...
I20260619 13:15:47.963019 388 mini_ranger.cc:68] Stopped Ranger
2026-06-19 13:15:47.963 UTC [11234] LOG: received smart shutdown request
2026-06-19 13:15:47.965 UTC [11234] LOG: background worker "logical replication launcher" (PID 11243) exited with exit code 1
2026-06-19 13:15:47.966 UTC [11238] LOG: shutting down
2026-06-19 13:15:48.097 UTC [11234] LOG: database system is shut down
2026-06-19T13:15:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46742 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:15:48 dist-test-slave-mkt3 krb5kdc[12229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:15:48 dist-test-slave-mkt3 krb5kdc[12229](info): set up 2 sockets
Jun 19 13:15:48 dist-test-slave-mkt3 krb5kdc[12229](info): commencing operation
krb5kdc: starting...
W20260619 13:15:49.229871 388 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:15:49 dist-test-slave-mkt3 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:15:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:15:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260619 13:15:49.504588 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwhPxxl/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/postgres -l logfile start
2026-06-19 13:15:51.592 UTC [12259] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 13:15:51.592 UTC [12259] LOG: listening on IPv4 address "127.0.97.20", port 43467
2026-06-19 13:15:51.595 UTC [12259] LOG: listening on Unix socket "/tmp/.s.PGSQL.43467"
2026-06-19 13:15:51.614 UTC [12262] LOG: database system was shut down at 2026-06-19 13:15:50 UTC
2026-06-19 13:15:51.617 UTC [12259] LOG: database system is ready to accept connections
I20260619 13:15:52.505668 388 mini_postgres.cc:96] Postgres bound to 43467
2026-06-19 13:15:52.521 UTC [12270] FATAL: database "slave" does not exist
127.0.97.20:43467 - accepting connections
I20260619 13:15:52.521589 388 mini_ranger.cc:161] Starting Ranger...
I20260619 13:15:52.544581 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 13:15:52.975183 388 mini_ranger.cc:87] Created ranger Postgres database
I20260619 13:15:52.975302 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-admin
2026-06-19 13:15:53,248 [I] DB FLAVOR :POSTGRES
2026-06-19 13:15:53,249 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 13:15:53,249 [I] Checking connection..
2026-06-19 13:15:53,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 13:15:53,502 [I] Checking connection passed.
2026-06-19 13:15:53,503 [I] --------- Verifying version history table ---------
2026-06-19 13:15:53,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:15:53,752 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 13:15:53,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:15:54,004 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 13:15:54,005 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 13:15:54,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 13:15:54.254 UTC [12334] WARNING: there is no transaction in progress
2026-06-19 13:15:54,259 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 13:15:54,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 13:15:54,506 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 13:15:54,506 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 13:15:54,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 13:15:54,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 13:15:54,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mkt3.c.gcp-upstream.internal','N') ;"
2026-06-19 13:15:55,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 13:15:55,467 [I] Table x_portal_user does not exist in database ranger
2026-06-19 13:15:55,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 13:15:55,705 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 13:15:55,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 13:15:55,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 13:15:56,180 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 13:15:56,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 13:15:57.086 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57.102 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57.117 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57.132 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57.149 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57.169 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57.230 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57.236 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57.249 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57.259 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57.268 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57.281 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57.290 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57.298 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57.304 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57.310 UTC [12442] WARNING: there is no transaction in progress
2026-06-19 13:15:57,871 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 13:15:57,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 13:15:58,108 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 13:15:58,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 13:15:58,361 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 13:15:58,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 13:15:58,593 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 13:15:58,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 13:15:58,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 13:15:59,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mkt3.c.gcp-upstream.internal';"
2026-06-19 13:15:59,308 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 13:15:59,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 13:15:59,541 [I] DB_PATCHES have already been applied
I20260619 13:15:59.547230 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwhPxxl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 13:15:59.547348 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260619 13:15:59.560245 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:34931
Jun 19, 2026 1:15:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:15:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:15:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 1:15:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 1:15:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 1:16:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 1:16:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 13:16:00.571903 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.050s user 0.001s sys 0.034s
Jun 19, 2026 1:16:18 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 1:16:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 1:16:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 1:16:18 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 1:16:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 1:16:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 1:16:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 1:16:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 1:16:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 1:16:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 1:16:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 1:16:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 1:16:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 1:16:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 1:16:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 1:16:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 1:16:18 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 1:16:19 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 13:16:20.288829 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 13:16:20 dist-test-slave-mkt3 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874980, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 13:16:20.420287 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 13:16:20.437389 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 13:16:21.107335 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 13:16:21.107430 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-kms
I20260619 13:16:21.107483 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:43467
2026-06-19 13:16:21,421 [I] DB FLAVOR :POSTGRES
2026-06-19 13:16:21,422 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 13:16:21,422 [I] Checking connection
2026-06-19 13:16:21,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 13:16:21,639 [I] connection success
2026-06-19 13:16:21,639 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 13:16:21,639 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 13:16:21,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 13:16:21,858 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 13:16:21,859 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 13:16:21,859 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 13:16:21,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:43467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 13:16:22,114 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 13:16:22.543602 388 mini_ranger_kms.cc:310] Created kms service
I20260619 13:16:22.700810 388 mini_ranger_kms.cc:326] Created kudu user
I20260619 13:16:22.745365 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 13:16:22 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874980, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:16:23.151367 388 mini_ranger_kms.cc:384] Added ranger policy
I20260619 13:16:23.151693 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwhPxxl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwhPxxl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 13:16:23.151762 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260619 13:16:23.167302 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12622
I20260619 13:16:23.167429 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:45563
Jun 19, 2026 1:16:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:16:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:16:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 1:16:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 1:16:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 1:16:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 13:16:25.193259 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.773s user 0.065s sys 0.264s
I20260619 13:16:25.193495 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 13:16:25.193636 388 mini_ranger_kms.cc:401] 127.0.97.20:45563/kms/v1/keys
Jun 19 13:16:25 dist-test-slave-mkt3 krb5kdc[12229](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781874985, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 13:16:25 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874985, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 13:16:26 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874985, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 13:16:28 dist-test-slave-mkt3 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:16:28.222193 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37125
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37125
--ranger_config_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45563/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:16:28.535781 12673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:28.536043 12673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:28.536170 12673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:28.543167 12673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:16:28.543260 12673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:28.543329 12673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:28.543396 12673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:16:28.543457 12673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:16:28.553776 12673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:45563/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37125
--ranger_config_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:28.555171 12673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:28.556910 12673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:28.571286 12679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:28.571405 12678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:28.571712 12681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:28.572018 12673 server_base.cc:1034] running on GCE node
I20260619 13:16:28.572978 12673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:28.574826 12673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:28.576094 12673 hybrid_clock.cc:648] HybridClock initialized: now 1781874988576051 us; error 37 us; skew 500 ppm
I20260619 13:16:28.576504 12673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:28 dist-test-slave-mkt3 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:28.579766 12673 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:16:28.581452 12673 webserver.cc:460] Webserver started at http://127.0.97.62:34621/ using document root <none> and password file <none>
I20260619 13:16:28.582252 12673 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:28.582366 12673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:28.582680 12673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:16:28 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:16:28.635478 12673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "e23f3a25d85944e2b80631913a55938b"
format_stamp: "Formatted at 2026-06-19 13:16:28 on dist-test-slave-mkt3"
server_key: "4f376755a67c50e08c2b4a2a26d2be49"
server_key_iv: "af147de70c353034427fd9cb20911630"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:28.636343 12673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "e23f3a25d85944e2b80631913a55938b"
format_stamp: "Formatted at 2026-06-19 13:16:28 on dist-test-slave-mkt3"
server_key: "4f376755a67c50e08c2b4a2a26d2be49"
server_key_iv: "af147de70c353034427fd9cb20911630"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:28.641777 12673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:16:28.661681 12688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:28.663267 12673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:16:28.663444 12673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "e23f3a25d85944e2b80631913a55938b"
format_stamp: "Formatted at 2026-06-19 13:16:28 on dist-test-slave-mkt3"
server_key: "4f376755a67c50e08c2b4a2a26d2be49"
server_key_iv: "af147de70c353034427fd9cb20911630"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:28.663647 12673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:28.679095 12673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:28.682475 12673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:28.682747 12673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:28.698848 12673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37125
I20260619 13:16:28.698886 12739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37125 every 8 connection(s)
I20260619 13:16:28.700067 12673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:16:28.703169 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 12673
I20260619 13:16:28.703393 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:16:28.703927 12740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:16:28.704187 388 external_mini_cluster.cc:1428] Setting key 651d4d7f8c567acaa60160000cf89463
I20260619 13:16:28.717540 12740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b: Bootstrap starting.
Jun 19 13:16:28 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:28.721230 12740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b: Neither blocks nor log segments found. Creating new log.
I20260619 13:16:28.722532 12740 log.cc:824] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:28.725647 12740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b: No bootstrap required, opened a new log
I20260619 13:16:28.726553 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:28.706108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.0.97.62:37125)
0619 13:16:28.706802 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:28.706816 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:28.706861 (+ 45us) server_negotiation.cc:407] Connection header received
0619 13:16:28.708521 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:28.708541 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:28.708922 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:28.709230 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:28.710781 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:28.712041 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:28.713748 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:28.714489 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:28.714613 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:28.718725 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:28.718735 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:28.718754 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:28.718802 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:28.722292 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:28.722970 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:28.722980 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:28.722989 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:28.723192 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:28.723584 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:28.723587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:28.723589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:28.724191 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:28.724495 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:28.724838 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 13:16:28.725214 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":327,"threads_started":1}
I20260619 13:16:28.732973 12740 raft_consensus.cc:348] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23f3a25d85944e2b80631913a55938b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37125 } }
I20260619 13:16:28.733295 12740 raft_consensus.cc:374] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:16:28.733382 12740 raft_consensus.cc:729] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e23f3a25d85944e2b80631913a55938b, State: Initialized, Role: FOLLOWER
I20260619 13:16:28.733873 12740 consensus_queue.cc:260] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23f3a25d85944e2b80631913a55938b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37125 } }
I20260619 13:16:28.734071 12740 raft_consensus.cc:388] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:16:28.734163 12740 raft_consensus.cc:482] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:16:28.734275 12740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:16:28.736533 12740 raft_consensus.cc:504] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23f3a25d85944e2b80631913a55938b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37125 } }
I20260619 13:16:28.736964 12740 leader_election.cc:302] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e23f3a25d85944e2b80631913a55938b; no voters:
I20260619 13:16:28.737427 12740 leader_election.cc:288] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:16:28.737550 12745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:16:28.738175 12745 raft_consensus.cc:686] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 1 LEADER]: Becoming Leader. State: Replica: e23f3a25d85944e2b80631913a55938b, State: Running, Role: LEADER
I20260619 13:16:28.738665 12745 consensus_queue.cc:237] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23f3a25d85944e2b80631913a55938b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37125 } }
I20260619 13:16:28.739495 12740 sys_catalog.cc:564] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:16:28.741819 12746 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e23f3a25d85944e2b80631913a55938b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23f3a25d85944e2b80631913a55938b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37125 } } }
I20260619 13:16:28.741829 12747 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e23f3a25d85944e2b80631913a55938b. Latest consensus state: current_term: 1 leader_uuid: "e23f3a25d85944e2b80631913a55938b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23f3a25d85944e2b80631913a55938b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37125 } } }
I20260619 13:16:28.742375 12746 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [sys.catalog]: This master's current role is: LEADER
I20260619 13:16:28.742375 12747 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [sys.catalog]: This master's current role is: LEADER
I20260619 13:16:28.746621 12753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:16:28.751679 12753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:16:28.752233 12740 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 13:16:28.767974 12753 catalog_manager.cc:1347] Generated new cluster ID: 954d99d54fc34adda17c4c4aa406e501
I20260619 13:16:28.768110 12753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:16:28.784736 12753 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:16:28.786645 12753 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:16:28.794101 12753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b: Generated new TSK 0
I20260619 13:16:28.795147 12753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:16:29.529250 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37125
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:16:29 dist-test-slave-mkt3 krb5kdc[12229](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781874989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 13:16:29.876834 12785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:29.877113 12785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:29.877251 12785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:29.884533 12785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:29.884660 12785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:29.884871 12785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:16:29.896122 12785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:45563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:29.897740 12785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:29.899402 12785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:29.914918 12795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:29.916422 12796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:29.920126 12785 server_base.cc:1034] running on GCE node
W20260619 13:16:29.920181 12798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:29.920827 12785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:29.921960 12785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:29.923262 12785 hybrid_clock.cc:648] HybridClock initialized: now 1781874989923187 us; error 59 us; skew 500 ppm
I20260619 13:16:29.923588 12785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:29 dist-test-slave-mkt3 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:29.927107 12785 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:16:29.929059 12785 webserver.cc:460] Webserver started at http://127.0.97.1:35089/ using document root <none> and password file <none>
I20260619 13:16:29.929971 12785 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:29.930115 12785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:29.930457 12785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:16:29 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:16:29.955612 12785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "2d32c21fb02c47969ae936e83b5e7b74"
format_stamp: "Formatted at 2026-06-19 13:16:29 on dist-test-slave-mkt3"
server_key: "a6d55059befec1f944a510d98813abde"
server_key_iv: "b8b883f08f08f2f952b9a35a0281afe9"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:29.956786 12785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2d32c21fb02c47969ae936e83b5e7b74"
format_stamp: "Formatted at 2026-06-19 13:16:29 on dist-test-slave-mkt3"
server_key: "a6d55059befec1f944a510d98813abde"
server_key_iv: "b8b883f08f08f2f952b9a35a0281afe9"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:29.964373 12785 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260619 13:16:29.995916 12805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:29.997952 12785 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.004s sys 0.001s
I20260619 13:16:29.998154 12785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "2d32c21fb02c47969ae936e83b5e7b74"
format_stamp: "Formatted at 2026-06-19 13:16:29 on dist-test-slave-mkt3"
server_key: "a6d55059befec1f944a510d98813abde"
server_key_iv: "b8b883f08f08f2f952b9a35a0281afe9"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:29.998426 12785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:30.018126 12785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:30.023504 12785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:30.023836 12785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:30.025698 12785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:16:30.033669 12785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:16:30.033859 12785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 13:16:30.034071 12785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:16:30.034205 12785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:30.130822 12785 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39079
I20260619 13:16:30.130864 12917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39079 every 8 connection(s)
I20260619 13:16:30.132604 12785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:16:30.138339 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 12785
I20260619 13:16:30.138567 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:16:30.139403 388 external_mini_cluster.cc:1428] Setting key 8cff7a7394d4ebd36e8f3af3a23981f4
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Jun 19 13:16:30 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:16:30.197046 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:30.136659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49205 (local address 127.0.97.62:37125)
0619 13:16:30.151997 (+ 15338us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:30.152005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:30.166512 (+ 14507us) server_negotiation.cc:407] Connection header received
0619 13:16:30.169314 (+ 2802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:30.169320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:30.169417 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:30.169497 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:30.171746 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:30.172800 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:30.175117 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:30.175740 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:30.175840 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:30.184903 (+ 9063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:30.184911 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:30.184916 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:30.184968 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:30.188272 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:30.192636 (+ 4364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:30.192641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:30.192645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:30.192807 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:30.193365 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:30.193370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:30.193374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:30.193719 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:30.193873 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:30.196444 (+ 2571us) server_negotiation.cc:299] Negotiation successful
0619 13:16:30.196678 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15165,"thread_start_us":15347,"threads_started":1}
I20260619 13:16:30.202131 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:30.137405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37125 (local address 127.0.97.1:49205)
0619 13:16:30.166304 (+ 28899us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:30.166369 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:30.168902 (+ 2533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:30.170108 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:30.170121 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:30.170750 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:30.171544 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:30.171569 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:30.173003 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:30.173008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:30.174959 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:30.174973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:30.176176 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:30.176183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:30.176449 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:30.180089 (+ 3640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:30.180112 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:30.184625 (+ 4513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:30.191830 (+ 7205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:30.191840 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:30.191858 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:30.192484 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:30.192966 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:30.192971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:30.192975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:30.193224 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:30.193965 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:30.193974 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:30.194340 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 13:16:30.199725 (+ 5385us) client_negotiation.cc:240] Negotiation successful
0619 13:16:30.200484 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28557,"thread_start_us":36338,"threads_started":1}
I20260619 13:16:30.204180 12918 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37125
I20260619 13:16:30.204509 12918 heartbeater.cc:464] Registering TS with master...
I20260619 13:16:30.205418 12918 heartbeater.cc:511] Master 127.0.97.62:37125 requested a full tablet report, sending...
I20260619 13:16:30.208141 12705 ts_manager.cc:194] Registered new tserver with Master: 2d32c21fb02c47969ae936e83b5e7b74 (127.0.97.1:39079)
I20260619 13:16:30.210253 12705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:16:30.220978 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37125
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:16:30 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 13:16:30 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 13:16:30.661687 12924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:30.662001 12924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:30.662138 12924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:30.671351 12924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:30.671535 12924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:30.671737 12924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:16:30.690089 12924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:45563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:30.702908 12924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:30.705236 12924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:30.750370 12931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:30.752667 12932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:30.758536 12924 server_base.cc:1034] running on GCE node
W20260619 13:16:30.762413 12935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:30.763346 12924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:30.780692 12924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:30.785236 12924 hybrid_clock.cc:648] HybridClock initialized: now 1781874990784509 us; error 743 us; skew 500 ppm
I20260619 13:16:30.785737 12924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:30 dist-test-slave-mkt3 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:30.791836 12924 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:16:30.812323 12924 webserver.cc:460] Webserver started at http://127.0.97.2:39375/ using document root <none> and password file <none>
I20260619 13:16:30.813376 12924 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:30.813604 12924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:30.814055 12924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:16:30 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:16:30.842198 12924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "56ba6804e79b4749b9fd8f0e75386bb9"
format_stamp: "Formatted at 2026-06-19 13:16:30 on dist-test-slave-mkt3"
server_key: "9d1d0294709f1f26000191de3d0641d3"
server_key_iv: "c0110ee4da8cbac497bc146fd172b6b4"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:30.843302 12924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "56ba6804e79b4749b9fd8f0e75386bb9"
format_stamp: "Formatted at 2026-06-19 13:16:30 on dist-test-slave-mkt3"
server_key: "9d1d0294709f1f26000191de3d0641d3"
server_key_iv: "c0110ee4da8cbac497bc146fd172b6b4"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:30.856295 12924 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.003s sys 0.008s
I20260619 13:16:30.911561 12941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:30.913498 12924 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.006s sys 0.001s
I20260619 13:16:30.913759 12924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "56ba6804e79b4749b9fd8f0e75386bb9"
format_stamp: "Formatted at 2026-06-19 13:16:30 on dist-test-slave-mkt3"
server_key: "9d1d0294709f1f26000191de3d0641d3"
server_key_iv: "c0110ee4da8cbac497bc146fd172b6b4"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:30.914131 12924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:30.970425 12924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:30.975840 12924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:30.976271 12924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:30.994114 12924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:16:30.999145 12924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:16:30.999329 12924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:30.999487 12924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:16:30.999575 12924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:31.214000 12918 heartbeater.cc:503] Master 127.0.97.62:37125 was elected leader, sending a full tablet report...
I20260619 13:16:31.229436 12924 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37217
I20260619 13:16:31.229462 13060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37217 every 8 connection(s)
I20260619 13:16:31.231244 12924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:16:31.233218 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 12924
I20260619 13:16:31.233377 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:16:31.233991 388 external_mini_cluster.cc:1428] Setting key b73728be5ab5350c2a2bbbf4172c6bf9
Jun 19 13:16:31 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:31.272109 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:31.234109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39573 (local address 127.0.97.62:37125)
0619 13:16:31.248717 (+ 14608us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:31.248723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:31.250697 (+ 1974us) server_negotiation.cc:407] Connection header received
0619 13:16:31.252901 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:31.252907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:31.252988 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:31.253082 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:31.255095 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:31.256089 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:31.258627 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:31.259216 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:31.259311 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:31.265193 (+ 5882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:31.265199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:31.265204 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:31.265243 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:31.268505 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:31.269383 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:31.269388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:31.269391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:31.269528 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:31.270043 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:31.270048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:31.270051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:31.270415 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:31.270539 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:31.271502 (+ 963us) server_negotiation.cc:299] Negotiation successful
0619 13:16:31.271743 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14467,"thread_start_us":14339,"threads_started":1}
I20260619 13:16:31.274415 13064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:31.234791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37125 (local address 127.0.97.2:39573)
0619 13:16:31.249286 (+ 14495us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:31.249345 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:31.252484 (+ 3139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:31.253497 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:31.253509 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:31.254100 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:31.254909 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:31.254934 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:31.256299 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:31.256303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:31.258470 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:31.258485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:31.259643 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:31.259649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:31.259937 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:31.262076 (+ 2139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:31.262116 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:31.264914 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:31.268660 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:31.268669 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:31.268688 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:31.269252 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:31.269660 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:31.269664 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:31.269668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:31.269924 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:31.270625 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:31.270633 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:31.271055 (+ 422us) client_negotiation.cc:769] Sending connection context
0619 13:16:31.272522 (+ 1467us) client_negotiation.cc:240] Negotiation successful
0619 13:16:31.273049 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14141,"thread_start_us":14939,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:16:31.276139 13061 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37125
I20260619 13:16:31.276450 13061 heartbeater.cc:464] Registering TS with master...
I20260619 13:16:31.277298 13061 heartbeater.cc:511] Master 127.0.97.62:37125 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 13:16:31.278915 12705 ts_manager.cc:194] Registered new tserver with Master: 56ba6804e79b4749b9fd8f0e75386bb9 (127.0.97.2:37217)
I20260619 13:16:31.279929 12705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:16:31.311180 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37125
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:16:31.634855 13067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:31.635099 13067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:31.635200 13067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:31.642169 13067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:31.642266 13067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:31.642377 13067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:16:31.651506 13067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:45563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:31.652789 13067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:31.654068 13067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:31.665731 13073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:31.665802 13075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:31.665773 13072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:31.666749 13067 server_base.cc:1034] running on GCE node
I20260619 13:16:31.667253 13067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:31.668125 13067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:31.669305 13067 hybrid_clock.cc:648] HybridClock initialized: now 1781874991669257 us; error 44 us; skew 500 ppm
I20260619 13:16:31.669569 13067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:31 dist-test-slave-mkt3 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:31.672308 13067 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:16:31.673571 13067 webserver.cc:460] Webserver started at http://127.0.97.3:37639/ using document root <none> and password file <none>
I20260619 13:16:31.674208 13067 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:31.674317 13067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:31.674578 13067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 13:16:31 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:16:31.689143 13067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "61fece2206d94ea2b48987c9c9276f09"
format_stamp: "Formatted at 2026-06-19 13:16:31 on dist-test-slave-mkt3"
server_key: "9b990ce0f77e186e09d4edb6523ae897"
server_key_iv: "d0795f87f911b9276d667cf6de273d18"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:31.690264 13067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "61fece2206d94ea2b48987c9c9276f09"
format_stamp: "Formatted at 2026-06-19 13:16:31 on dist-test-slave-mkt3"
server_key: "9b990ce0f77e186e09d4edb6523ae897"
server_key_iv: "d0795f87f911b9276d667cf6de273d18"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:31.695889 13067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:16:31.716132 13082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:31.717814 13067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:16:31.718007 13067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "61fece2206d94ea2b48987c9c9276f09"
format_stamp: "Formatted at 2026-06-19 13:16:31 on dist-test-slave-mkt3"
server_key: "9b990ce0f77e186e09d4edb6523ae897"
server_key_iv: "d0795f87f911b9276d667cf6de273d18"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:31.718257 13067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:31.738981 13067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:31.742522 13067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:31.742746 13067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:31.743528 13067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:16:31.744706 13067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:16:31.744773 13067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:31.744858 13067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:16:31.744899 13067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:31.772711 13067 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41303
I20260619 13:16:31.772728 13194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41303 every 8 connection(s)
I20260619 13:16:31.773909 13067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:16:31.777171 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 13067
I20260619 13:16:31.777344 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:16:31.777912 388 external_mini_cluster.cc:1428] Setting key b1b326cadd54324423fec79c7810c2bd
Jun 19 13:16:31 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:31.792239 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:31.776387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34619 (local address 127.0.97.62:37125)
0619 13:16:31.776876 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:31.776883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:31.777884 (+ 1001us) server_negotiation.cc:407] Connection header received
0619 13:16:31.779918 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:31.779924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:31.780005 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:31.780094 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:31.781502 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:31.782271 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:31.783713 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:31.784120 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:31.784225 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:31.787170 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:31.787174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:31.787177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:31.787207 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:31.789374 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:31.790042 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:31.790046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:31.790048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:31.790163 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:31.790577 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:31.790580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:31.790582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:31.790827 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:31.790963 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:31.791669 (+ 706us) server_negotiation.cc:299] Negotiation successful
0619 13:16:31.791892 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":229,"threads_started":1}
I20260619 13:16:31.793202 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:31.776887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37125 (local address 127.0.97.3:34619)
0619 13:16:31.777700 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:31.777750 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:31.779662 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:31.780335 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:31.780344 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:31.780727 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:31.781352 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:31.781369 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:31.782413 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:31.782417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:31.783586 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:31.783597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:31.784333 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:31.784341 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:31.784550 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:31.785088 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:31.785109 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:31.786986 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:31.789504 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:31.789512 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:31.789529 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:31.789935 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:31.790259 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:31.790264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:31.790268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:31.790466 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:31.790937 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:31.790945 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:31.791283 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 13:16:31.791552 (+ 269us) client_negotiation.cc:240] Negotiation successful
0619 13:16:31.791999 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":318,"threads_started":1}
I20260619 13:16:31.794380 13195 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37125
I20260619 13:16:31.794662 13195 heartbeater.cc:464] Registering TS with master...
I20260619 13:16:31.795318 13195 heartbeater.cc:511] Master 127.0.97.62:37125 requested a full tablet report, sending...
I20260619 13:16:31.796604 12705 ts_manager.cc:194] Registered new tserver with Master: 61fece2206d94ea2b48987c9c9276f09 (127.0.97.3:41303)
I20260619 13:16:31.797380 12705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34619
I20260619 13:16:31.805034 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:16:31.806677 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 12785
Jun 19 13:16:31 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:16:31.821791 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:31.807495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52070 (local address 127.0.97.1:39079)
0619 13:16:31.808036 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:31.808043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:31.808066 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:16:31.808128 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:31.808133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:31.808311 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:31.808431 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:31.809871 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:31.810899 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:31.812213 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:31.812758 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:31.812849 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:31.816696 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:31.816706 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:31.816719 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:31.816750 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:31.819414 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:31.819984 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:31.819992 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:31.819994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:31.820139 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:31.820449 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:31.820452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:31.820454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:31.820765 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:31.821009 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:31.821219 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 13:16:31.821437 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":284,"threads_started":1}
I20260619 13:16:31.822979 12913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:31.902938 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 12785
I20260619 13:16:31.915465 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 12924
Jun 19 13:16:31 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:16:31.930573 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:31.916291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.0.97.2:37217)
0619 13:16:31.916783 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:31.916790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:31.916816 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:16:31.916876 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:31.916882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:31.917084 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:31.917218 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:31.918446 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:31.919473 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:31.920918 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:31.921462 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:31.921582 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:31.925467 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:31.925477 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:31.925491 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:31.925531 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:31.927994 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:31.928579 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:31.928586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:31.928588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:31.928723 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:31.929054 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:31.929058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:31.929060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:31.929391 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:31.929674 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:31.929998 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 13:16:31.930251 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":246,"threads_started":1}
I20260619 13:16:31.931752 13056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:32.012202 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 12924
I20260619 13:16:32.024729 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 13067
Jun 19 13:16:32 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:16:32.040598 13204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:32.025622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57098 (local address 127.0.97.3:41303)
0619 13:16:32.026195 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:32.026202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:32.026230 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:16:32.026288 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:32.026295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:32.026501 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:32.026622 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:32.027741 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:32.028714 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:32.030024 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:32.030555 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:32.030649 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:32.035115 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:32.035131 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:32.035147 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:32.035207 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:32.038130 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:32.038814 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:32.038821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:32.038823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:32.038946 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:32.039230 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:32.039232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:32.039234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:32.039544 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:32.039777 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:32.040113 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 13:16:32.040308 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":285,"threads_started":1}
I20260619 13:16:32.041903 13190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:32.121614 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 13067
I20260619 13:16:32.133787 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 12673
I20260619 13:16:32.135013 12735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:32.237380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 12673
I20260619 13:16:32.254009 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37125
--webserver_interface=127.0.97.62
--webserver_port=34621
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37125
--ranger_config_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45563/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45563/kms with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:16:32.559495 13207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:32.559796 13207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:32.559926 13207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:32.566895 13207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:16:32.567009 13207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:32.567085 13207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:32.567167 13207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:16:32.567231 13207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:16:32.576725 13207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45563/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37125
--ranger_config_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:32.577935 13207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:32.579190 13207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:32.590153 13213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:32.590102 13215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:32.590186 13212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:32.590651 13207 server_base.cc:1034] running on GCE node
I20260619 13:16:32.591413 13207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:32.592743 13207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:32.593917 13207 hybrid_clock.cc:648] HybridClock initialized: now 1781874992593868 us; error 45 us; skew 500 ppm
I20260619 13:16:32.594233 13207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:32 dist-test-slave-mkt3 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:32.597147 13207 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:16:32.598548 13207 webserver.cc:460] Webserver started at http://127.0.97.62:34621/ using document root <none> and password file <none>
I20260619 13:16:32.599156 13207 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:32.599272 13207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 13:16:32 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:16:32.651485 13207 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20260619 13:16:32.654467 13223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:32.655747 13207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:16:32.655921 13207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "e23f3a25d85944e2b80631913a55938b"
format_stamp: "Formatted at 2026-06-19 13:16:28 on dist-test-slave-mkt3"
server_key: "4f376755a67c50e08c2b4a2a26d2be49"
server_key_iv: "af147de70c353034427fd9cb20911630"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:32.656872 13207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:32.676074 13207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:32.679364 13207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:32.679610 13207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:32.696442 13207 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37125
I20260619 13:16:32.696462 13274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37125 every 8 connection(s)
I20260619 13:16:32.697795 13207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:16:32.702754 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 13207
I20260619 13:16:32.703735 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39079
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37125
--webserver_port=35089
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45563/kms with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:16:32.704864 13275 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:16:32.709101 13275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b: Bootstrap starting.
I20260619 13:16:32.727036 13275 log.cc:824] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:32.735168 13275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:16:32.735637 13275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b: Bootstrap complete.
I20260619 13:16:32.744081 13275 raft_consensus.cc:348] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23f3a25d85944e2b80631913a55938b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37125 } }
I20260619 13:16:32.744864 13275 raft_consensus.cc:729] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e23f3a25d85944e2b80631913a55938b, State: Initialized, Role: FOLLOWER
I20260619 13:16:32.745416 13275 consensus_queue.cc:260] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23f3a25d85944e2b80631913a55938b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37125 } }
I20260619 13:16:32.745604 13275 raft_consensus.cc:388] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:16:32.745704 13275 raft_consensus.cc:482] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:16:32.745880 13275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:16:32.749416 13275 raft_consensus.cc:504] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23f3a25d85944e2b80631913a55938b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37125 } }
I20260619 13:16:32.749927 13275 leader_election.cc:302] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e23f3a25d85944e2b80631913a55938b; no voters:
I20260619 13:16:32.750491 13275 leader_election.cc:288] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 13:16:32.750624 13279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 2 FOLLOWER]: Leader election won for term 2
I20260619 13:16:32.751709 13279 raft_consensus.cc:686] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [term 2 LEADER]: Becoming Leader. State: Replica: e23f3a25d85944e2b80631913a55938b, State: Running, Role: LEADER
I20260619 13:16:32.752516 13275 sys_catalog.cc:564] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:16:32.752445 13279 consensus_queue.cc:237] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23f3a25d85944e2b80631913a55938b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37125 } }
I20260619 13:16:32.755012 13280 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e23f3a25d85944e2b80631913a55938b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23f3a25d85944e2b80631913a55938b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37125 } } }
I20260619 13:16:32.755374 13280 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [sys.catalog]: This master's current role is: LEADER
I20260619 13:16:32.755007 13281 sys_catalog.cc:455] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e23f3a25d85944e2b80631913a55938b. Latest consensus state: current_term: 2 leader_uuid: "e23f3a25d85944e2b80631913a55938b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e23f3a25d85944e2b80631913a55938b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37125 } } }
I20260619 13:16:32.755645 13281 sys_catalog.cc:458] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b [sys.catalog]: This master's current role is: LEADER
I20260619 13:16:32.763077 13285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:16:32.768668 13285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:16:32.770329 13285 catalog_manager.cc:1259] Loaded cluster ID: 954d99d54fc34adda17c4c4aa406e501
I20260619 13:16:32.770491 13285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:16:32.774556 13285 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:16:32.776057 13285 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b: Loaded TSK: 0
I20260619 13:16:32.777698 13285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:16:32.780481 13275 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 13:16:33.124298 13277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:33.124807 13277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:33.125023 13277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:33.136231 13277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:33.136380 13277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:33.136572 13277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:16:33.151685 13277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35089
--tserver_master_addrs=127.0.97.62:37125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:33.153396 13277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:33.155200 13277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:33.199003 13313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:33.200296 13312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:33.201581 13277 server_base.cc:1034] running on GCE node
W20260619 13:16:33.201445 13315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:33.202570 13277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:33.204224 13277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:33.208079 13277 hybrid_clock.cc:648] HybridClock initialized: now 1781874993208028 us; error 39 us; skew 500 ppm
I20260619 13:16:33.208496 13277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:33 dist-test-slave-mkt3 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:33.212869 13277 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:16:33.231921 13277 webserver.cc:460] Webserver started at http://127.0.97.1:35089/ using document root <none> and password file <none>
I20260619 13:16:33.233035 13277 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:33.233283 13277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 13:16:33 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:16:33.255344 13277 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:16:33.258749 13324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:33.264190 13277 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260619 13:16:33.264381 13277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "2d32c21fb02c47969ae936e83b5e7b74"
format_stamp: "Formatted at 2026-06-19 13:16:29 on dist-test-slave-mkt3"
server_key: "a6d55059befec1f944a510d98813abde"
server_key_iv: "b8b883f08f08f2f952b9a35a0281afe9"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:33.265722 13277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:33.325580 13277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:33.329509 13277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:33.329773 13277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:33.330857 13277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:16:33.332285 13277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:16:33.332387 13277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:33.332490 13277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:16:33.332558 13277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:33.379442 13277 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39079
I20260619 13:16:33.379506 13436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39079 every 8 connection(s)
I20260619 13:16:33.381373 13277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:16:33.386305 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 13277
I20260619 13:16:33.387019 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37217
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37125
--webserver_port=39375
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45563/kms with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:16:33 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:33.420481 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:33.384949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54639 (local address 127.0.97.62:37125)
0619 13:16:33.398770 (+ 13821us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:33.398787 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:33.398834 (+ 47us) server_negotiation.cc:407] Connection header received
0619 13:16:33.399263 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:33.399284 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:33.399779 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:33.400190 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:33.401948 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:33.403761 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:33.405904 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:33.406870 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:33.407023 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:33.411595 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:33.411606 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:33.411628 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:33.411674 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:33.415594 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:33.416510 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:33.416519 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:33.416528 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:33.416751 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:33.417252 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:33.417258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:33.417262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:33.417939 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:33.418108 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:33.419184 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0619 13:16:33.419521 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13572,"thread_start_us":13345,"threads_started":1}
I20260619 13:16:33.420635 13440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:33.385001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37125 (local address 127.0.97.1:54639)
0619 13:16:33.394716 (+ 9715us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:33.394770 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:33.397151 (+ 2381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:33.400335 (+ 3184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:33.400345 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:33.400932 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:33.401683 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:33.401710 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:33.403949 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:33.403954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:33.405742 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:33.405757 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:33.407399 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:33.407405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:33.407678 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:33.408423 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:33.408460 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:33.411243 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:33.415765 (+ 4522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:33.415775 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:33.415792 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:33.416352 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:33.416867 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:33.416871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:33.416875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:33.417118 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:33.418195 (+ 1077us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:33.418204 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:33.418604 (+ 400us) client_negotiation.cc:769] Sending connection context
0619 13:16:33.418917 (+ 313us) client_negotiation.cc:240] Negotiation successful
0619 13:16:33.419444 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9392,"thread_start_us":12469,"threads_started":1}
I20260619 13:16:33.424196 13437 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37125
I20260619 13:16:33.424481 13437 heartbeater.cc:464] Registering TS with master...
I20260619 13:16:33.425235 13437 heartbeater.cc:511] Master 127.0.97.62:37125 requested a full tablet report, sending...
W20260619 13:16:33.428390 13437 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37125 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 13:16:33.428612 13437 heartbeater.cc:464] Registering TS with master...
I20260619 13:16:33.428872 13437 heartbeater.cc:511] Master 127.0.97.62:37125 requested a full tablet report, sending...
I20260619 13:16:33.430264 13437 heartbeater.cc:464] Registering TS with master...
I20260619 13:16:33.430511 13437 heartbeater.cc:511] Master 127.0.97.62:37125 requested a full tablet report, sending...
W20260619 13:16:33.431573 13437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:16:33.756034 13441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:33.756343 13441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:33.756475 13441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:33.767503 13441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:33.767637 13441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:33.767869 13441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:16:33.781683 13441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=39375
--tserver_master_addrs=127.0.97.62:37125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:33.783308 13441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:33.784906 13441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:33.799587 13456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:33.800146 13455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:33.803548 13441 server_base.cc:1034] running on GCE node
W20260619 13:16:33.804435 13458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:33.805194 13441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:33.813540 13441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:33.817286 13441 hybrid_clock.cc:648] HybridClock initialized: now 1781874993817206 us; error 94 us; skew 500 ppm
I20260619 13:16:33.817770 13441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:33 dist-test-slave-mkt3 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:33.827076 13441 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:16:33.830986 13441 webserver.cc:460] Webserver started at http://127.0.97.2:39375/ using document root <none> and password file <none>
I20260619 13:16:33.831911 13441 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:33.832060 13441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 13:16:33 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:16:33.866039 13441 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:16:33.869725 13468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:33.871583 13441 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 13:16:33.871783 13441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "56ba6804e79b4749b9fd8f0e75386bb9"
format_stamp: "Formatted at 2026-06-19 13:16:30 on dist-test-slave-mkt3"
server_key: "9d1d0294709f1f26000191de3d0641d3"
server_key_iv: "c0110ee4da8cbac497bc146fd172b6b4"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:33.873147 13441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:33.904382 13441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:33.909482 13441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:33.909862 13441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:33.911103 13441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:16:33.912962 13441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:16:33.913110 13441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:33.913252 13441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:16:33.913358 13441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 13:16:33 dist-test-slave-mkt3 krb5kdc[12229](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781874993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:33.986462 13441 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37217
I20260619 13:16:33.986502 13581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37217 every 8 connection(s)
I20260619 13:16:33.987970 13441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:16:33.989162 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 13441
I20260619 13:16:33.990034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:45563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41303
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37125
--webserver_port=37639
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45563/kms with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:16:34 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:34.035477 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:33.997479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37125 (local address 127.0.97.2:58307)
0619 13:16:34.006729 (+ 9250us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:34.006802 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:34.009237 (+ 2435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:34.018646 (+ 9409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:34.018661 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:34.019288 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:34.019926 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:34.019946 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.021134 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.021138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:34.022588 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:34.022601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.023379 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.023386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:34.023608 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:34.024227 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:34.024254 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:34.026561 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:34.030572 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:34.030580 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:34.030598 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:34.031214 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:34.031662 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:34.031667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:34.031671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:34.031921 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:34.032749 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:34.032758 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:34.033175 (+ 417us) client_negotiation.cc:769] Sending connection context
0619 13:16:34.033473 (+ 298us) client_negotiation.cc:240] Negotiation successful
0619 13:16:34.034050 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8890,"thread_start_us":8651,"threads_started":1}
I20260619 13:16:34.036518 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:33.998139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58307 (local address 127.0.97.62:37125)
0619 13:16:34.011980 (+ 13841us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:34.011986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:34.012009 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:16:34.012076 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:34.012081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:34.012173 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:34.012298 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:34.020156 (+ 7858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.020947 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.022732 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.023148 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.023263 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:34.026846 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:34.026854 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:34.026859 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:34.026902 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:34.030408 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:34.031363 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:34.031368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:34.031372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:34.031523 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:34.032052 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:34.032058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:34.032061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:34.032394 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:34.032692 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:34.036122 (+ 3430us) server_negotiation.cc:299] Negotiation successful
0619 13:16:34.036287 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13701,"thread_start_us":14280,"threads_started":1}
I20260619 13:16:34.037693 13582 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37125
I20260619 13:16:34.037995 13582 heartbeater.cc:464] Registering TS with master...
I20260619 13:16:34.038697 13582 heartbeater.cc:511] Master 127.0.97.62:37125 requested a full tablet report, sending...
I20260619 13:16:34.040349 13240 ts_manager.cc:194] Registered new tserver with Master: 56ba6804e79b4749b9fd8f0e75386bb9 (127.0.97.2:37217)
I20260619 13:16:34.041966 13240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58307
Jun 19 13:16:34 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 13:16:34.402621 13583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:34.402981 13583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:34.403132 13583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:34.414320 13583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:34.414515 13583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:34.414747 13583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:16:34.430699 13583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:45563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37639
--tserver_master_addrs=127.0.97.62:37125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:34.431725 13437 heartbeater.cc:464] Registering TS with master...
I20260619 13:16:34.432123 13437 heartbeater.cc:511] Master 127.0.97.62:37125 requested a full tablet report, sending...
I20260619 13:16:34.432595 13583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:34.433341 13240 ts_manager.cc:194] Registered new tserver with Master: 2d32c21fb02c47969ae936e83b5e7b74 (127.0.97.1:39079)
I20260619 13:16:34.434289 13583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 13:16:34.434485 13240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54639
W20260619 13:16:34.478518 13591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:34.478543 13592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:34.482532 13594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:34.482988 13583 server_base.cc:1034] running on GCE node
I20260619 13:16:34.483615 13583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:34.484745 13583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:34.486038 13583 hybrid_clock.cc:648] HybridClock initialized: now 1781874994485971 us; error 57 us; skew 500 ppm
I20260619 13:16:34.486356 13583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:34 dist-test-slave-mkt3 krb5kdc[12229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:34.489395 13583 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:16:34.494767 13583 webserver.cc:460] Webserver started at http://127.0.97.3:37639/ using document root <none> and password file <none>
I20260619 13:16:34.495450 13583 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:34.495579 13583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 13:16:34 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:16:34.516023 13583 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:16:34.519315 13602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:34.520925 13583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:16:34.521111 13583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "61fece2206d94ea2b48987c9c9276f09"
format_stamp: "Formatted at 2026-06-19 13:16:31 on dist-test-slave-mkt3"
server_key: "9b990ce0f77e186e09d4edb6523ae897"
server_key_iv: "d0795f87f911b9276d667cf6de273d18"
server_key_version: "kuduclusterkey@0"
I20260619 13:16:34.522455 13583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:34.556216 13583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:34.560014 13583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:34.560257 13583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:34.561299 13583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:16:34.562772 13583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:16:34.562871 13583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:34.562969 13583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:16:34.563045 13583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:34.610800 13583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41303
I20260619 13:16:34.610841 13714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41303 every 8 connection(s)
I20260619 13:16:34.612272 13583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:16:34.615883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 13583
Jun 19 13:16:34 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:34.649181 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:34.617531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.0.97.62:37125)
0619 13:16:34.628536 (+ 11005us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:34.628542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:34.628561 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:16:34.628625 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:34.628630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:34.628714 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:34.628828 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:34.633396 (+ 4568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.634543 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.637196 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.637815 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.637965 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:34.641339 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:34.641345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:34.641350 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:34.641387 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:34.644894 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:34.645689 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:34.645694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:34.645697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:34.645867 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:34.646537 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:34.646542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:34.646549 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:34.646924 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:34.648454 (+ 1530us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:34.648592 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 13:16:34.648807 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10884,"spinlock_wait_cycles":2432,"thread_start_us":10726,"threads_started":1}
I20260619 13:16:34.654580 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:34.615224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43305 (local address 127.0.97.62:37125)
0619 13:16:34.626241 (+ 11017us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:34.626247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:34.632944 (+ 6697us) server_negotiation.cc:407] Connection header received
0619 13:16:34.633004 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:34.633009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:34.633097 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:34.633207 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:34.635877 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.637027 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.640184 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.640839 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.640959 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:34.645221 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:34.645228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:34.645233 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:34.645268 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:34.648562 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:34.649437 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:34.649442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:34.649446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:34.649578 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:34.650125 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:34.650130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:34.650134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:34.650572 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:34.650720 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:34.654019 (+ 3299us) server_negotiation.cc:299] Negotiation successful
0619 13:16:34.654217 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10898,"spinlock_wait_cycles":4864,"thread_start_us":10767,"threads_started":1}
I20260619 13:16:34.655476 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:34.615879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37125 (local address 127.0.97.3:43305)
0619 13:16:34.630172 (+ 14293us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:34.630219 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:34.632595 (+ 2376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:34.633526 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:34.633539 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:34.634161 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:34.635644 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:34.635672 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.637190 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.637195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:34.639997 (+ 2802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:34.640021 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.641277 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.641285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:34.641565 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:34.642288 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:34.642317 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:34.644950 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:34.648728 (+ 3778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:34.648737 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:34.648754 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:34.649298 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:34.649727 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:34.649732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:34.649736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:34.649982 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:34.650878 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:34.650893 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:34.652335 (+ 1442us) client_negotiation.cc:769] Sending connection context
0619 13:16:34.652647 (+ 312us) client_negotiation.cc:240] Negotiation successful
0619 13:16:34.653160 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12165,"spinlock_wait_cycles":307456,"thread_start_us":12951,"threads_started":1}
I20260619 13:16:34.656960 13715 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37125
I20260619 13:16:34.657264 13715 heartbeater.cc:464] Registering TS with master...
I20260619 13:16:34.658129 13715 heartbeater.cc:511] Master 127.0.97.62:37125 requested a full tablet report, sending...
I20260619 13:16:34.659708 13240 ts_manager.cc:194] Registered new tserver with Master: 61fece2206d94ea2b48987c9c9276f09 (127.0.97.3:41303)
I20260619 13:16:34.660832 13240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43305
I20260619 13:16:34.665385 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:16:34.688807 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:34.668908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.0.97.62:37125)
0619 13:16:34.669061 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:34.669067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:34.673798 (+ 4731us) server_negotiation.cc:407] Connection header received
0619 13:16:34.673971 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:34.673976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:34.674063 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:34.674160 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:34.676196 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.677259 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.678917 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.679522 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.679608 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:34.682612 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:34.682618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:34.682622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:34.682654 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:34.685640 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:34.686497 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:34.686503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:34.686506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:34.686645 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:34.687084 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:34.687089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:34.687092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:34.687462 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:34.687682 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:34.688257 (+ 575us) server_negotiation.cc:299] Negotiation successful
0619 13:16:34.688473 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:16:34.695238 13240 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:16:34.706799 13240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Jun 19 13:16:34 dist-test-slave-mkt3 krb5kdc[12229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 13:16:34.751746 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:34.729649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39079 (local address 127.0.0.1:49264)
0619 13:16:34.731548 (+ 1899us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:34.731581 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:34.731756 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:34.743365 (+ 11609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:34.743375 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:34.743411 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:16:34.743892 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:34.743906 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.746192 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.746199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:34.748781 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:34.748799 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.750824 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.750831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:34.751021 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:34.751083 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 13:16:34.751188 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 13:16:34.751381 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1638,"thread_start_us":1503,"threads_started":1}
I20260619 13:16:34.753180 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:34.730166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49264 (local address 127.0.97.1:39079)
0619 13:16:34.742429 (+ 12263us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:34.742440 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:34.742473 (+ 33us) server_negotiation.cc:407] Connection header received
0619 13:16:34.742602 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:34.742611 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:34.742905 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:34.744122 (+ 1217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:16:34.744371 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.745953 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.748998 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.750568 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.752270 (+ 1702us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:34.752531 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:34.752605 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 13:16:34.752792 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12060,"thread_start_us":12901,"threads_started":1}
I20260619 13:16:34.757500 13372 tablet_service.cc:1467] Processing CreateTablet for tablet 2bba23827db645479651c28ffa0aada2 (DEFAULT_TABLE table=test-table [id=1512236ce82a4f9c992e49b83d85d8c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:16:34.759155 13372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bba23827db645479651c28ffa0aada2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:16:34.779245 13737 tablet_bootstrap.cc:492] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74: Bootstrap starting.
I20260619 13:16:34.784812 13737 tablet_bootstrap.cc:654] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74: Neither blocks nor log segments found. Creating new log.
I20260619 13:16:34.786352 13737 log.cc:824] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:34.787560 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:34.730823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37217 (local address 127.0.0.1:47084)
0619 13:16:34.751988 (+ 21165us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:34.752006 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:34.752102 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:34.776768 (+ 24666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:34.776774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:34.776809 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:16:34.777008 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:34.777021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.782768 (+ 5747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.782775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:34.784586 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:34.784602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.786736 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.786742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:34.786939 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:34.786984 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 13:16:34.787062 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 13:16:34.787238 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21057,"thread_start_us":47192,"threads_started":1}
I20260619 13:16:34.788697 13736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:34.734108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47084 (local address 127.0.97.2:37217)
0619 13:16:34.764113 (+ 30005us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:34.764123 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:34.764163 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:16:34.764257 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:34.764265 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:34.764495 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:34.777230 (+ 12735us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:16:34.777489 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.780853 (+ 3364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.784786 (+ 3933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.786416 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.787885 (+ 1469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:34.788111 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:34.788196 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 13:16:34.788371 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29840,"thread_start_us":30513,"threads_started":1}
I20260619 13:16:34.792493 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:34.731377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41303 (local address 127.0.0.1:52790)
0619 13:16:34.779082 (+ 47705us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:34.779103 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:34.779230 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:34.784196 (+ 4966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:34.784201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:34.784228 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:16:34.784593 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:34.784607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.786425 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.786431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:34.788386 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:34.788402 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.791663 (+ 3261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.791669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:34.791841 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:34.791874 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:16:34.791963 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 13:16:34.792097 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46780,"mutex_wait_us":1,"thread_start_us":51122,"threads_started":1}
I20260619 13:16:34.792649 13516 tablet_service.cc:1467] Processing CreateTablet for tablet 2bba23827db645479651c28ffa0aada2 (DEFAULT_TABLE table=test-table [id=1512236ce82a4f9c992e49b83d85d8c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:16:34.793548 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:34.732130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52790 (local address 127.0.97.3:41303)
0619 13:16:34.783209 (+ 51079us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:34.783217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:34.783249 (+ 32us) server_negotiation.cc:407] Connection header received
0619 13:16:34.783344 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:34.783351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:34.783602 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:34.783877 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:16:34.784821 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.786289 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.788563 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:34.791182 (+ 2619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:34.791307 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:34.791519 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:34.792709 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0619 13:16:34.792867 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50900,"thread_start_us":50796,"threads_started":1}
I20260619 13:16:34.794164 13516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bba23827db645479651c28ffa0aada2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:16:34.797187 13650 tablet_service.cc:1467] Processing CreateTablet for tablet 2bba23827db645479651c28ffa0aada2 (DEFAULT_TABLE table=test-table [id=1512236ce82a4f9c992e49b83d85d8c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:16:34.798673 13650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bba23827db645479651c28ffa0aada2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:16:34.801226 13737 tablet_bootstrap.cc:492] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74: No bootstrap required, opened a new log
I20260619 13:16:34.801553 13737 ts_tablet_manager.cc:1397] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.011s
I20260619 13:16:34.811987 13737 raft_consensus.cc:348] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d32c21fb02c47969ae936e83b5e7b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39079 } } peers { permanent_uuid: "56ba6804e79b4749b9fd8f0e75386bb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37217 } } peers { permanent_uuid: "61fece2206d94ea2b48987c9c9276f09" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260619 13:16:34.812470 13737 raft_consensus.cc:374] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:16:34.812605 13737 raft_consensus.cc:729] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d32c21fb02c47969ae936e83b5e7b74, State: Initialized, Role: FOLLOWER
I20260619 13:16:34.813534 13737 consensus_queue.cc:260] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d32c21fb02c47969ae936e83b5e7b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39079 } } peers { permanent_uuid: "56ba6804e79b4749b9fd8f0e75386bb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37217 } } peers { permanent_uuid: "61fece2206d94ea2b48987c9c9276f09" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260619 13:16:34.817634 13739 tablet_bootstrap.cc:492] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9: Bootstrap starting.
I20260619 13:16:34.819314 13737 ts_tablet_manager.cc:1428] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74: Time spent starting tablet: real 0.018s user 0.008s sys 0.007s
I20260619 13:16:34.819653 13437 heartbeater.cc:503] Master 127.0.97.62:37125 was elected leader, sending a full tablet report...
I20260619 13:16:34.826880 13741 tablet_bootstrap.cc:492] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09: Bootstrap starting.
I20260619 13:16:34.827272 13739 tablet_bootstrap.cc:654] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9: Neither blocks nor log segments found. Creating new log.
I20260619 13:16:34.828749 13739 log.cc:824] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:34.832337 13741 tablet_bootstrap.cc:654] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09: Neither blocks nor log segments found. Creating new log.
I20260619 13:16:34.832801 13739 tablet_bootstrap.cc:492] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9: No bootstrap required, opened a new log
I20260619 13:16:34.833086 13739 ts_tablet_manager.cc:1397] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260619 13:16:34.833549 13741 log.cc:824] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:34.843541 13739 raft_consensus.cc:348] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d32c21fb02c47969ae936e83b5e7b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39079 } } peers { permanent_uuid: "56ba6804e79b4749b9fd8f0e75386bb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37217 } } peers { permanent_uuid: "61fece2206d94ea2b48987c9c9276f09" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260619 13:16:34.844007 13739 raft_consensus.cc:374] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:16:34.844107 13739 raft_consensus.cc:729] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ba6804e79b4749b9fd8f0e75386bb9, State: Initialized, Role: FOLLOWER
I20260619 13:16:34.844859 13739 consensus_queue.cc:260] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d32c21fb02c47969ae936e83b5e7b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39079 } } peers { permanent_uuid: "56ba6804e79b4749b9fd8f0e75386bb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37217 } } peers { permanent_uuid: "61fece2206d94ea2b48987c9c9276f09" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260619 13:16:34.846345 13741 tablet_bootstrap.cc:492] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09: No bootstrap required, opened a new log
I20260619 13:16:34.846696 13741 ts_tablet_manager.cc:1397] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.008s
I20260619 13:16:34.852025 13582 heartbeater.cc:503] Master 127.0.97.62:37125 was elected leader, sending a full tablet report...
I20260619 13:16:34.853121 13739 ts_tablet_manager.cc:1428] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9: Time spent starting tablet: real 0.020s user 0.013s sys 0.008s
I20260619 13:16:34.854910 13741 raft_consensus.cc:348] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d32c21fb02c47969ae936e83b5e7b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39079 } } peers { permanent_uuid: "56ba6804e79b4749b9fd8f0e75386bb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37217 } } peers { permanent_uuid: "61fece2206d94ea2b48987c9c9276f09" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260619 13:16:34.855371 13741 raft_consensus.cc:374] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:16:34.855513 13741 raft_consensus.cc:729] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61fece2206d94ea2b48987c9c9276f09, State: Initialized, Role: FOLLOWER
I20260619 13:16:34.856256 13741 consensus_queue.cc:260] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d32c21fb02c47969ae936e83b5e7b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39079 } } peers { permanent_uuid: "56ba6804e79b4749b9fd8f0e75386bb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37217 } } peers { permanent_uuid: "61fece2206d94ea2b48987c9c9276f09" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260619 13:16:34.857373 13715 heartbeater.cc:503] Master 127.0.97.62:37125 was elected leader, sending a full tablet report...
I20260619 13:16:34.858062 13741 ts_tablet_manager.cc:1428] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
W20260619 13:16:34.885576 13438 tablet.cc:2369] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:16:34.886191 13716 tablet.cc:2369] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:16:34.999400 13744 raft_consensus.cc:482] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:16:34.999728 13744 raft_consensus.cc:504] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d32c21fb02c47969ae936e83b5e7b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39079 } } peers { permanent_uuid: "56ba6804e79b4749b9fd8f0e75386bb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37217 } } peers { permanent_uuid: "61fece2206d94ea2b48987c9c9276f09" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260619 13:16:35.004552 13744 leader_election.cc:288] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d32c21fb02c47969ae936e83b5e7b74 (127.0.97.1:39079), 61fece2206d94ea2b48987c9c9276f09 (127.0.97.3:41303)
W20260619 13:16:35.014127 13586 tablet.cc:2369] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:16:35.017663 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:35.003581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39471 (local address 127.0.97.1:39079)
0619 13:16:35.004117 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:35.004123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:35.010204 (+ 6081us) server_negotiation.cc:407] Connection header received
0619 13:16:35.010540 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:35.010546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:35.010643 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:35.010759 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:16:35.011534 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.012632 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.015240 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.016401 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.016495 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:35.016602 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:35.017299 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 13:16:35.017425 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376}
I20260619 13:16:35.017699 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:35.003881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37425 (local address 127.0.97.3:41303)
0619 13:16:35.004453 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:35.004458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:35.010482 (+ 6024us) server_negotiation.cc:407] Connection header received
0619 13:16:35.010726 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:35.010743 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:35.010819 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:35.010882 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:16:35.011718 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.012726 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.015227 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.016396 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.016495 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:35.016602 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:35.017299 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 13:16:35.017455 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260619 13:16:35.018054 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:35.002925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41303 (local address 127.0.97.2:37425)
0619 13:16:35.010395 (+ 7470us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:35.010409 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:35.010572 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:35.011110 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:35.011114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:35.011144 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:16:35.011547 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:35.011560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.012849 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.012855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:35.015052 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:35.015068 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.016829 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.016834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:35.017070 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:35.017142 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 13:16:35.017229 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 13:16:35.017353 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7364,"mutex_wait_us":66,"thread_start_us":6787,"threads_started":1}
I20260619 13:16:35.018631 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:35.002530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39079 (local address 127.0.97.2:39471)
0619 13:16:35.010054 (+ 7524us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:35.010081 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:35.010351 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:35.010937 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:35.010942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:35.010967 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:16:35.011353 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:35.011368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.012739 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.012744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:35.015077 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:35.015086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.016807 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.016817 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:35.017067 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:35.017136 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 13:16:35.017941 (+ 805us) client_negotiation.cc:240] Negotiation successful
0619 13:16:35.018069 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7383,"mutex_wait_us":101,"thread_start_us":7577,"threads_started":1}
I20260619 13:16:35.019104 13670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bba23827db645479651c28ffa0aada2" candidate_uuid: "56ba6804e79b4749b9fd8f0e75386bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61fece2206d94ea2b48987c9c9276f09" is_pre_election: true
I20260619 13:16:35.019496 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bba23827db645479651c28ffa0aada2" candidate_uuid: "56ba6804e79b4749b9fd8f0e75386bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d32c21fb02c47969ae936e83b5e7b74" is_pre_election: true
I20260619 13:16:35.019654 13670 raft_consensus.cc:2454] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56ba6804e79b4749b9fd8f0e75386bb9 in term 0.
I20260619 13:16:35.020017 13392 raft_consensus.cc:2454] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56ba6804e79b4749b9fd8f0e75386bb9 in term 0.
I20260619 13:16:35.020388 13472 leader_election.cc:302] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56ba6804e79b4749b9fd8f0e75386bb9, 61fece2206d94ea2b48987c9c9276f09; no voters:
I20260619 13:16:35.020751 13744 raft_consensus.cc:2781] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:16:35.020869 13744 raft_consensus.cc:482] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:16:35.020964 13744 raft_consensus.cc:3037] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:16:35.023104 13744 raft_consensus.cc:504] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d32c21fb02c47969ae936e83b5e7b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39079 } } peers { permanent_uuid: "56ba6804e79b4749b9fd8f0e75386bb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37217 } } peers { permanent_uuid: "61fece2206d94ea2b48987c9c9276f09" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260619 13:16:35.023670 13744 leader_election.cc:288] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [CANDIDATE]: Term 1 election: Requested vote from peers 2d32c21fb02c47969ae936e83b5e7b74 (127.0.97.1:39079), 61fece2206d94ea2b48987c9c9276f09 (127.0.97.3:41303)
I20260619 13:16:35.024096 13392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bba23827db645479651c28ffa0aada2" candidate_uuid: "56ba6804e79b4749b9fd8f0e75386bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d32c21fb02c47969ae936e83b5e7b74"
I20260619 13:16:35.024287 13392 raft_consensus.cc:3037] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:16:35.024269 13670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bba23827db645479651c28ffa0aada2" candidate_uuid: "56ba6804e79b4749b9fd8f0e75386bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61fece2206d94ea2b48987c9c9276f09"
I20260619 13:16:35.024456 13670 raft_consensus.cc:3037] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:16:35.026528 13392 raft_consensus.cc:2454] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56ba6804e79b4749b9fd8f0e75386bb9 in term 1.
I20260619 13:16:35.027171 13469 leader_election.cc:302] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d32c21fb02c47969ae936e83b5e7b74, 56ba6804e79b4749b9fd8f0e75386bb9; no voters:
I20260619 13:16:35.027354 13670 raft_consensus.cc:2454] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56ba6804e79b4749b9fd8f0e75386bb9 in term 1.
I20260619 13:16:35.027652 13744 raft_consensus.cc:2781] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:16:35.028223 13744 raft_consensus.cc:686] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [term 1 LEADER]: Becoming Leader. State: Replica: 56ba6804e79b4749b9fd8f0e75386bb9, State: Running, Role: LEADER
I20260619 13:16:35.028720 13744 consensus_queue.cc:237] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d32c21fb02c47969ae936e83b5e7b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39079 } } peers { permanent_uuid: "56ba6804e79b4749b9fd8f0e75386bb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37217 } } peers { permanent_uuid: "61fece2206d94ea2b48987c9c9276f09" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260619 13:16:35.033525 13239 catalog_manager.cc:5495] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56ba6804e79b4749b9fd8f0e75386bb9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "56ba6804e79b4749b9fd8f0e75386bb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d32c21fb02c47969ae936e83b5e7b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56ba6804e79b4749b9fd8f0e75386bb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61fece2206d94ea2b48987c9c9276f09" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } health_report { overall_health: UNKNOWN } } }
I20260619 13:16:35.112237 13736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:35.106738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47086 (local address 127.0.97.2:37217)
0619 13:16:35.106935 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:35.106940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:35.106959 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:16:35.107084 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:35.107088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:35.107189 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:35.107315 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:16:35.107696 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.108422 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.110183 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.110575 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.110683 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:35.111166 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:16:35.111669 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:16:35.111798 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:35.111900 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 13:16:35.111980 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:16:35.121446 13670 raft_consensus.cc:1261] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09 [term 1 FOLLOWER]: Refusing update from remote peer 56ba6804e79b4749b9fd8f0e75386bb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:16:35.121567 13392 raft_consensus.cc:1261] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74 [term 1 FOLLOWER]: Refusing update from remote peer 56ba6804e79b4749b9fd8f0e75386bb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:16:35.122437 13744 consensus_queue.cc:1035] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61fece2206d94ea2b48987c9c9276f09" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:16:35.122748 13755 consensus_queue.cc:1035] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d32c21fb02c47969ae936e83b5e7b74" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:16:35.163311 13239 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:16:35.163587 13239 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:16:35.167732 13239 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e23f3a25d85944e2b80631913a55938b: Sending DeleteTablet for 3 replicas of tablet 2bba23827db645479651c28ffa0aada2
I20260619 13:16:35.169270 13516 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bba23827db645479651c28ffa0aada2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:16:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47084
I20260619 13:16:35.169376 13372 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bba23827db645479651c28ffa0aada2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:16:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49264
I20260619 13:16:35.169399 13650 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bba23827db645479651c28ffa0aada2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:16:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52790
I20260619 13:16:35.170817 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 13277
I20260619 13:16:35.186221 13770 tablet_replica.cc:330] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09: stopping tablet replica
I20260619 13:16:35.186405 13769 tablet_replica.cc:330] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74: stopping tablet replica
I20260619 13:16:35.186676 13770 raft_consensus.cc:2229] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:16:35.186823 13769 raft_consensus.cc:2229] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:16:35.187186 13770 raft_consensus.cc:2258] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:16:35.187305 13769 raft_consensus.cc:2258] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:16:35.189342 13770 ts_tablet_manager.cc:1905] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:16:35.194309 13768 tablet_replica.cc:330] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9: stopping tablet replica
I20260619 13:16:35.194710 13768 raft_consensus.cc:2229] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:16:35.195288 13768 raft_consensus.cc:2258] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:16:35.196184 13770 ts_tablet_manager.cc:1918] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:16:35.196337 13770 log.cc:1196] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/2bba23827db645479651c28ffa0aada2
I20260619 13:16:35.197167 13770 ts_tablet_manager.cc:1939] T 2bba23827db645479651c28ffa0aada2 P 61fece2206d94ea2b48987c9c9276f09: Deleting consensus metadata
I20260619 13:16:35.197217 13768 ts_tablet_manager.cc:1905] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:16:35.198315 13769 ts_tablet_manager.cc:1905] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:16:35.198813 13227 catalog_manager.cc:4841] TS 61fece2206d94ea2b48987c9c9276f09 (127.0.97.3:41303): tablet 2bba23827db645479651c28ffa0aada2 (table test-table [id=1512236ce82a4f9c992e49b83d85d8c6]) successfully deleted
I20260619 13:16:35.203661 13768 ts_tablet_manager.cc:1918] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:16:35.203812 13768 log.cc:1196] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/2bba23827db645479651c28ffa0aada2
I20260619 13:16:35.204305 13768 ts_tablet_manager.cc:1939] T 2bba23827db645479651c28ffa0aada2 P 56ba6804e79b4749b9fd8f0e75386bb9: Deleting consensus metadata
I20260619 13:16:35.205593 13769 ts_tablet_manager.cc:1918] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:16:35.205773 13769 log.cc:1196] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/2bba23827db645479651c28ffa0aada2
I20260619 13:16:35.206316 13769 ts_tablet_manager.cc:1939] T 2bba23827db645479651c28ffa0aada2 P 2d32c21fb02c47969ae936e83b5e7b74: Deleting consensus metadata
I20260619 13:16:35.207998 13224 catalog_manager.cc:4841] TS 2d32c21fb02c47969ae936e83b5e7b74 (127.0.97.1:39079): tablet 2bba23827db645479651c28ffa0aada2 (table test-table [id=1512236ce82a4f9c992e49b83d85d8c6]) successfully deleted
I20260619 13:16:35.209497 13225 catalog_manager.cc:4841] TS 56ba6804e79b4749b9fd8f0e75386bb9 (127.0.97.2:37217): tablet 2bba23827db645479651c28ffa0aada2 (table test-table [id=1512236ce82a4f9c992e49b83d85d8c6]) successfully deleted
I20260619 13:16:35.210865 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:35.173940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.0.97.1:39079)
0619 13:16:35.174120 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:35.174126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:35.182688 (+ 8562us) server_negotiation.cc:407] Connection header received
0619 13:16:35.182862 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:35.182867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:35.182958 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:35.183049 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:35.187421 (+ 4372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.188581 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.198801 (+ 10220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.199418 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.199517 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:35.203062 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:35.203079 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:35.203097 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:35.203152 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:35.207194 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:35.208108 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:35.208117 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:35.208121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:35.208312 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:35.208784 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:35.208789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:35.208793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:35.209242 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:35.209928 (+ 686us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:35.210093 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 13:16:35.210307 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:16:35.211515 13432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:35.354821 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 13277
I20260619 13:16:35.368044 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 13441
I20260619 13:16:35.381630 13736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:35.368747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.0.97.2:37217)
0619 13:16:35.368943 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:35.368949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:35.368967 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:16:35.369121 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:35.369126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:35.369207 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:35.369306 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:35.370644 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.371465 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.372691 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.373142 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.373262 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:35.375698 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:35.375711 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:35.375727 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:35.375761 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:35.378907 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:35.379616 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:35.379625 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:35.379627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:35.379773 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:35.380115 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:35.380120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:35.380123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:35.380474 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:35.380770 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:35.381061 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 13:16:35.381294 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 13:16:35.382302 13576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:35.494783 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 13441
I20260619 13:16:35.509526 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 13583
I20260619 13:16:35.521798 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:35.510254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.0.97.3:41303)
0619 13:16:35.510429 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:35.510435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:35.510459 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:16:35.510635 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:35.510641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:35.510707 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:35.510788 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:35.511999 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.512694 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.513956 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:35.514350 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:35.514454 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:35.516788 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:35.516800 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:35.516817 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:35.516847 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:35.519290 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:35.519918 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:35.519925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:35.519927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:35.520056 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:35.520385 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:35.520390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:35.520393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:35.520725 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:35.520980 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:35.521286 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 13:16:35.521490 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:16:35.522514 13710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:35.607508 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 13583
I20260619 13:16:35.620127 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 13207
I20260619 13:16:35.621616 13270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:35.730686 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 13207
I20260619 13:16:35.748045 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 13:16:36.082393 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 13:16:36.082610 388 mini_ranger.cc:66] Stopping Ranger...
I20260619 13:16:36.416917 388 mini_ranger.cc:68] Stopped Ranger
2026-06-19 13:16:36.417 UTC [12259] LOG: received smart shutdown request
2026-06-19 13:16:36.419 UTC [12259] LOG: background worker "logical replication launcher" (PID 12268) exited with exit code 1
2026-06-19 13:16:36.420 UTC [12263] LOG: shutting down
2026-06-19 13:16:36.544 UTC [12259] LOG: database system is shut down
2026-06-19T13:16:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48444 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:16:36 dist-test-slave-mkt3 krb5kdc[13784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:16:36 dist-test-slave-mkt3 krb5kdc[13784](info): set up 2 sockets
Jun 19 13:16:36 dist-test-slave-mkt3 krb5kdc[13784](info): commencing operation
krb5kdc: starting...
W20260619 13:16:37.641775 388 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:16:37 dist-test-slave-mkt3 krb5kdc[13784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:16:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:16:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:16:37.833269 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37141
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37141,127.0.97.61:39565,127.0.97.60:36787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:16:38.140735 13797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:38.141027 13797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:38.141168 13797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:38.148209 13797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:16:38.148324 13797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:38.148413 13797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:38.148490 13797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:16:38.148547 13797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:16:38.158237 13797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37141,127.0.97.61:39565,127.0.97.60:36787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:38.159539 13797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:38.160825 13797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:38.171725 13802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:38.171648 13805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:38.171695 13803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:38.172719 13797 server_base.cc:1034] running on GCE node
I20260619 13:16:38.173406 13797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:38.174757 13797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:38.175942 13797 hybrid_clock.cc:648] HybridClock initialized: now 1781874998175900 us; error 33 us; skew 500 ppm
I20260619 13:16:38.176261 13797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:38 dist-test-slave-mkt3 krb5kdc[13784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:38.179179 13797 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:16:38.180858 13797 webserver.cc:460] Webserver started at http://127.0.97.62:43423/ using document root <none> and password file <none>
I20260619 13:16:38.181519 13797 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:38.181629 13797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:38.181941 13797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:16:38.184738 13797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "d207bf922fea41e5a63b4093fb87b475"
format_stamp: "Formatted at 2026-06-19 13:16:38 on dist-test-slave-mkt3"
server_key: "571474d3584716836bede632e1b42f25"
server_key_iv: "82d1fb16be281dca1b272934647d4dee"
server_key_version: "encryptionkey@0"
I20260619 13:16:38.185487 13797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "d207bf922fea41e5a63b4093fb87b475"
format_stamp: "Formatted at 2026-06-19 13:16:38 on dist-test-slave-mkt3"
server_key: "571474d3584716836bede632e1b42f25"
server_key_iv: "82d1fb16be281dca1b272934647d4dee"
server_key_version: "encryptionkey@0"
I20260619 13:16:38.190313 13797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260619 13:16:38.193608 13812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:38.194906 13797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:16:38.195057 13797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "d207bf922fea41e5a63b4093fb87b475"
format_stamp: "Formatted at 2026-06-19 13:16:38 on dist-test-slave-mkt3"
server_key: "571474d3584716836bede632e1b42f25"
server_key_iv: "82d1fb16be281dca1b272934647d4dee"
server_key_version: "encryptionkey@0"
I20260619 13:16:38.195230 13797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:38.214924 13797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:38.218756 13797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:38.219007 13797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:38.235725 13797 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37141
I20260619 13:16:38.235754 13863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37141 every 8 connection(s)
I20260619 13:16:38.236970 13797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:16:38.240509 13864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:16:38.241006 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 13797
I20260619 13:16:38.241205 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:16:38.241765 388 external_mini_cluster.cc:1428] Setting key 7d3e5ef9726d3ca941c7cc18cb9e050f
I20260619 13:16:38.246712 13864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 19 13:16:38 dist-test-slave-mkt3 krb5kdc[13784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:38.269621 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.249658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.0.97.62:37141)
0619 13:16:38.251257 (+ 1599us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:38.251264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:38.251312 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:16:38.252779 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:38.252785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:38.253011 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:38.253195 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:38.254501 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.255749 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.257092 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.257647 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.257822 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:38.262269 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:38.262281 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:38.262299 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:38.262344 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:38.265609 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:38.266353 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:38.266362 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:38.266366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:38.266552 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:38.266944 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:38.266949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:38.266952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:38.267587 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:38.267787 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:38.268259 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 13:16:38.268543 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":1392,"spinlock_wait_cycles":17152,"thread_start_us":1194,"threads_started":1}
I20260619 13:16:38.269660 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.249657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.0.1:60714)
0619 13:16:38.250470 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.250525 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:38.252416 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:38.253219 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:38.253226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:38.253978 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:38.254314 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:38.254332 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.255889 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.255893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:38.256957 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:38.256969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.257897 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.257906 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:38.258126 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:38.258747 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:38.258774 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:38.262015 (+ 3241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:38.265758 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:38.265768 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:38.265779 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:38.266209 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:38.266661 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:38.266664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:38.266667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:38.266842 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:38.267715 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:38.267722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:38.267898 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 13:16:38.268132 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 13:16:38.268688 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":107,"thread_start_us":359,"threads_started":1}
I20260619 13:16:38.272765 13864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:16:38.273794 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.273400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39565 (local address 127.0.0.1:49344)
0619 13:16:38.273560 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.273663 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 13:16:38.274091 13816 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
I20260619 13:16:38.275518 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.275187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39565 (local address 127.0.0.1:49356)
0619 13:16:38.275327 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.275403 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 13:16:38.275693 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39565: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 13:16:38.288620 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39565
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37141,127.0.97.61:39565,127.0.97.60:36787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:16:38.325006 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } attempt: 1
I20260619 13:16:38.326097 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.325640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39565 (local address 127.0.0.1:49368)
0619 13:16:38.325834 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.325967 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 13:16:38.327164 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.326796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39565 (local address 127.0.0.1:49380)
0619 13:16:38.326970 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.327035 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 13:16:38.327353 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39565: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
I20260619 13:16:38.395583 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } attempt: 2
I20260619 13:16:38.396659 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.396239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39565 (local address 127.0.0.1:49394)
0619 13:16:38.396433 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.396528 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 13:16:38.397715 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.397362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39565 (local address 127.0.0.1:49406)
0619 13:16:38.397513 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.397580 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 13:16:38.397912 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39565: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
I20260619 13:16:38.489144 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } attempt: 3
I20260619 13:16:38.490361 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.489908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39565 (local address 127.0.0.1:49418)
0619 13:16:38.490114 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.490226 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 13:16:38.491395 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.491014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39565 (local address 127.0.0.1:49428)
0619 13:16:38.491166 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.491253 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 13:16:38.491586 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39565: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
W20260619 13:16:38.595290 13871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:38.595575 13871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:38.595712 13871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:38.605425 13871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:16:38.605549 13871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:38.605629 13871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:38.605716 13871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:16:38.605934 13871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:16:38.619926 13871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37141,127.0.97.61:39565,127.0.97.60:36787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:38.621408 13871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:38.623162 13871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 13:16:38.634930 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } attempt: 4
I20260619 13:16:38.636368 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.635683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39565 (local address 127.0.0.1:49432)
0619 13:16:38.636086 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.636186 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259}
W20260619 13:16:38.637465 13877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:38.637753 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.637151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39565 (local address 127.0.0.1:49442)
0619 13:16:38.637423 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.637565 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260619 13:16:38.637995 13876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:38.637991 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39565: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39565: connect: Connection refused (error 111)
W20260619 13:16:38.638304 13879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:38.638537 13871 server_base.cc:1034] running on GCE node
I20260619 13:16:38.639453 13871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:38.640972 13871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:38.642151 13871 hybrid_clock.cc:648] HybridClock initialized: now 1781874998642096 us; error 49 us; skew 500 ppm
I20260619 13:16:38.642478 13871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:38 dist-test-slave-mkt3 krb5kdc[13784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:38.645179 13871 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 13:16:38.646548 13871 webserver.cc:460] Webserver started at http://127.0.97.61:43973/ using document root <none> and password file <none>
I20260619 13:16:38.647156 13871 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:38.647271 13871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:38.647531 13871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:16:38.650312 13871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/data/instance:
uuid: "900594647ab54e26b7dd5d1288b14bad"
format_stamp: "Formatted at 2026-06-19 13:16:38 on dist-test-slave-mkt3"
server_key: "5f56261e41d9c46703e9c7b427460951"
server_key_iv: "0900bea4e6e495bbe08e9a7ceb8e4d78"
server_key_version: "encryptionkey@0"
I20260619 13:16:38.651015 13871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/wal/instance:
uuid: "900594647ab54e26b7dd5d1288b14bad"
format_stamp: "Formatted at 2026-06-19 13:16:38 on dist-test-slave-mkt3"
server_key: "5f56261e41d9c46703e9c7b427460951"
server_key_iv: "0900bea4e6e495bbe08e9a7ceb8e4d78"
server_key_version: "encryptionkey@0"
I20260619 13:16:38.655813 13871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:16:38.658919 13886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:38.660145 13871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:16:38.660310 13871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/wal
uuid: "900594647ab54e26b7dd5d1288b14bad"
format_stamp: "Formatted at 2026-06-19 13:16:38 on dist-test-slave-mkt3"
server_key: "5f56261e41d9c46703e9c7b427460951"
server_key_iv: "0900bea4e6e495bbe08e9a7ceb8e4d78"
server_key_version: "encryptionkey@0"
I20260619 13:16:38.660473 13871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:38.688699 13871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:38.692449 13871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:38.692696 13871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:38.709208 13871 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39565
I20260619 13:16:38.709263 13937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39565 every 8 connection(s)
I20260619 13:16:38.710484 13871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/data/info.pb
I20260619 13:16:38.713944 13938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:16:38.717784 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 13871
I20260619 13:16:38.717964 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-1/wal/instance
I20260619 13:16:38.718341 388 external_mini_cluster.cc:1428] Setting key 757c0c346bf3ee4d29c3ed9e0d6c237b
I20260619 13:16:38.720068 13938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:16:38 dist-test-slave-mkt3 krb5kdc[13784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260619 13:16:38.739446 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.722517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60718 (local address 127.0.97.62:37141)
0619 13:16:38.722710 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:38.722716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:38.723809 (+ 1093us) server_negotiation.cc:407] Connection header received
0619 13:16:38.725619 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:38.725624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:38.725700 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:38.725809 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:38.727217 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.728219 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.729638 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.730233 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.730351 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:38.733472 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:38.733479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:38.733484 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:38.733525 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:38.736455 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:38.737178 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:38.737181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:38.737184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:38.737296 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:38.737734 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:38.737739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:38.737743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:38.738066 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:38.738196 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:38.738863 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 13:16:38.739081 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:16:38.740314 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.722909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.0.1:60718)
0619 13:16:38.723633 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.723686 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:38.725354 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:38.726031 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:38.726039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:38.726520 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:38.727060 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:38.727078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.728344 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.728347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:38.729514 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:38.729526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.730425 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.730430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:38.730603 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:38.731255 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:38.731277 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:38.733252 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:38.736606 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:38.736612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:38.736624 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:38.737075 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:38.737379 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:38.737382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:38.737384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:38.737632 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:38.738193 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:38.738200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:38.738509 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 13:16:38.738741 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 13:16:38.739205 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":303,"threads_started":1}
I20260619 13:16:38.741933 13938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:16:38 dist-test-slave-mkt3 krb5kdc[13784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:16:38.757097 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.742719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39565 (local address 127.0.0.1:49452)
0619 13:16:38.742878 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.742899 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:38.742994 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:38.743929 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:38.743934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:38.744352 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:38.744528 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:38.744539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.746459 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.746462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:38.747249 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:38.747259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.748323 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.748328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:38.748459 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:38.748996 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:38.749014 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:38.751063 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:38.754821 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:38.754825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:38.754827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:38.755190 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:38.755586 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:38.755590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:38.755592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:38.755742 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:38.756411 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:38.756414 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:38.756522 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 13:16:38.756628 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 13:16:38.756807 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 13:16:38.757366 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.742801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49452 (local address 127.0.97.61:39565)
0619 13:16:38.743491 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:38.743499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:38.743527 (+ 28us) server_negotiation.cc:407] Connection header received
0619 13:16:38.743584 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:38.743592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:38.743785 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:38.743898 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:38.744714 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.746300 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.747375 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.748145 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.748257 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:38.751273 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:38.751286 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:38.751302 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:38.751337 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:38.754680 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:38.755315 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:38.755324 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:38.755327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:38.755463 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:38.755853 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:38.755859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:38.755862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:38.756287 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:38.756501 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:38.756774 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 13:16:38.756951 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":392,"threads_started":1}
I20260619 13:16:38.759649 13938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:16:38.760767 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.760317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60438)
0619 13:16:38.760488 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.760645 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 13:16:38.761027 13888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 13:16:38.762485 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36787
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37141,127.0.97.61:39565,127.0.97.60:36787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:16:38.762563 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.762198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60446)
0619 13:16:38.762351 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.762444 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 13:16:38.762828 13938 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36787: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
I20260619 13:16:38.812150 13938 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } attempt: 1
I20260619 13:16:38.813295 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.812755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60448)
0619 13:16:38.812981 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.813111 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 13:16:38.814354 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.813974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60458)
0619 13:16:38.814114 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.814219 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 13:16:38.814570 13938 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36787: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
I20260619 13:16:38.882786 13938 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } attempt: 2
I20260619 13:16:38.883913 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.883375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60470)
0619 13:16:38.883596 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.883712 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 13:16:38.884996 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.884598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60480)
0619 13:16:38.884755 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.884854 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 13:16:38.885243 13938 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36787: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
I20260619 13:16:38.937222 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } attempt: 5
Jun 19 13:16:38 dist-test-slave-mkt3 krb5kdc[13784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:16:38.964457 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.937888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39565 (local address 127.0.0.1:49468)
0619 13:16:38.938124 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.938149 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:38.938296 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:38.938754 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:38.938760 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:38.939317 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:38.939583 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:38.939598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.940628 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.940634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:38.942112 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:38.942131 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.942887 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.942893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:38.943089 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:38.955493 (+ 12404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:38.955523 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:38.958431 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:38.962129 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:38.962135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:38.962138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:38.962498 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:38.962891 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:38.962894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:38.962896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:38.963017 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:38.963683 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:38.963686 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:38.963813 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 13:16:38.963924 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 13:16:38.964122 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 13:16:38.964603 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.938018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49468 (local address 127.0.97.61:39565)
0619 13:16:38.938194 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:38.938199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:38.938218 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:16:38.938472 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:38.938477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:38.938563 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:38.938668 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:38.939759 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.940457 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.942269 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:38.942727 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:38.942825 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:38.958722 (+ 15897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:38.958730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:38.958734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:38.958782 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:38.961973 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:38.962634 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:38.962640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:38.962644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:38.962791 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:38.963131 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:38.963137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:38.963140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:38.963567 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:38.963731 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:38.964076 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 13:16:38.964303 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:16:38.965734 13864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:16:38.966795 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.966430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60486)
0619 13:16:38.966586 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.966671 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 13:16:38.967854 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.967446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60488)
0619 13:16:38.967641 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.967746 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 13:16:38.968045 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36787: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
I20260619 13:16:38.976468 13938 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } attempt: 3
I20260619 13:16:38.977440 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.976979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60502)
0619 13:16:38.977166 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.977280 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 13:16:38.978500 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:38.978062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60504)
0619 13:16:38.978296 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:38.978380 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260619 13:16:38.978704 13938 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36787: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
I20260619 13:16:39.019277 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } attempt: 1
I20260619 13:16:39.020397 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.019946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60514)
0619 13:16:39.020142 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:39.020233 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 13:16:39.021341 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.021013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60518)
0619 13:16:39.021164 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:39.021236 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 13:16:39.021526 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36787: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
W20260619 13:16:39.071666 13945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:39.071945 13945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:39.072082 13945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:39.079072 13945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:16:39.079182 13945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:39.079259 13945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:39.079339 13945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:16:39.079397 13945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:16:39.088920 13945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37141,127.0.97.61:39565,127.0.97.60:36787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:39.089713 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } attempt: 2
I20260619 13:16:39.090265 13945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:39.090794 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.090321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60522)
0619 13:16:39.090532 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:39.090631 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 13:16:39.091588 13945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 13:16:39.091684 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.091370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60528)
0619 13:16:39.091504 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:39.091571 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 13:16:39.091923 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36787: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
W20260619 13:16:39.102654 13951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:39.102768 13953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:39.102769 13950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:39.103576 13945 server_base.cc:1034] running on GCE node
I20260619 13:16:39.104327 13945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:39.105679 13945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:39.106884 13945 hybrid_clock.cc:648] HybridClock initialized: now 1781874999106822 us; error 48 us; skew 500 ppm
I20260619 13:16:39.107203 13945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:39 dist-test-slave-mkt3 krb5kdc[13784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:39.110096 13945 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 13:16:39.111451 13945 webserver.cc:460] Webserver started at http://127.0.97.60:33839/ using document root <none> and password file <none>
I20260619 13:16:39.112115 13945 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:39.112228 13945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:39.112504 13945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:16:39.115460 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/data/instance:
uuid: "2fa254b53392447d9df7266bb25b43e8"
format_stamp: "Formatted at 2026-06-19 13:16:39 on dist-test-slave-mkt3"
server_key: "60bd5dff78003360f522aa87644a1982"
server_key_iv: "c876771e54233b8ddf280ed9985d3c52"
server_key_version: "encryptionkey@0"
I20260619 13:16:39.116222 13945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/wal/instance:
uuid: "2fa254b53392447d9df7266bb25b43e8"
format_stamp: "Formatted at 2026-06-19 13:16:39 on dist-test-slave-mkt3"
server_key: "60bd5dff78003360f522aa87644a1982"
server_key_iv: "c876771e54233b8ddf280ed9985d3c52"
server_key_version: "encryptionkey@0"
I20260619 13:16:39.121081 13945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:16:39.121928 13938 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } attempt: 4
I20260619 13:16:39.122975 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.122543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60540)
0619 13:16:39.122755 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:39.122837 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 13:16:39.123972 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.123641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60542)
0619 13:16:39.123774 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:39.123848 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 13:16:39.124212 13938 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36787: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36787: connect: Connection refused (error 111)
I20260619 13:16:39.124828 13960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:39.126185 13945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:16:39.126329 13945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/wal
uuid: "2fa254b53392447d9df7266bb25b43e8"
format_stamp: "Formatted at 2026-06-19 13:16:39 on dist-test-slave-mkt3"
server_key: "60bd5dff78003360f522aa87644a1982"
server_key_iv: "c876771e54233b8ddf280ed9985d3c52"
server_key_version: "encryptionkey@0"
I20260619 13:16:39.126503 13945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:39.150048 13945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:39.153980 13945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:39.154227 13945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:39.171056 13945 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36787
I20260619 13:16:39.171087 14011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36787 every 8 connection(s)
I20260619 13:16:39.172291 13945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/data/info.pb
I20260619 13:16:39.175805 14012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:16:39.179780 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 13945
I20260619 13:16:39.179949 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781874807295266-388-0/minicluster-data/master-2/wal/instance
I20260619 13:16:39.180511 388 external_mini_cluster.cc:1428] Setting key 4a9777d5522a194adf0880ad4e6033a8
I20260619 13:16:39.182978 14012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:16:39 dist-test-slave-mkt3 krb5kdc[13784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 19 13:16:39 dist-test-slave-mkt3 krb5kdc[13784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 13:16:39.198115 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } attempt: 3
I20260619 13:16:39.201562 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.186732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.0.97.62:37141)
0619 13:16:39.186969 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:39.186976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:39.187416 (+ 440us) server_negotiation.cc:407] Connection header received
0619 13:16:39.187591 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:39.187595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:39.187655 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:39.187731 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:39.188812 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.189474 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.191040 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.191537 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.191655 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.195154 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:39.195159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:39.195163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:39.195194 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:39.197594 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.199099 (+ 1505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.199104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.199107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.199272 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.199751 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.199757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.199761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.200057 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:39.200174 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:39.201008 (+ 834us) server_negotiation.cc:299] Negotiation successful
0619 13:16:39.201223 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:16:39.202903 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.186658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37141 (local address 127.0.0.1:60732)
0619 13:16:39.187295 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:39.187326 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:39.187468 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:39.187812 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:39.187818 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:39.188385 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:39.188661 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:39.188672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.189612 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.189617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:39.190899 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:39.190914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.191785 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.191792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:39.191960 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.192484 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:39.192517 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:39.194974 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:39.197716 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:39.197727 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:39.197742 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:39.198321 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:39.199436 (+ 1115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:39.199441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:39.199444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:39.199609 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:39.200180 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:39.200188 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:39.200619 (+ 431us) client_negotiation.cc:769] Sending connection context
0619 13:16:39.200889 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 13:16:39.201388 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":32,"thread_start_us":326,"threads_started":1}
I20260619 13:16:39.202922 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.182352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.0.97.60:36787)
0619 13:16:39.183092 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:39.183108 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:39.183152 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:16:39.184818 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:39.184846 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:39.185249 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:39.185548 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:39.187190 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.188613 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.190147 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.190827 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.190949 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.195596 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:39.195603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:39.195616 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:39.195648 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:39.198553 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.199371 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.199379 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.199382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.199574 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.200712 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.200716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.200719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.201162 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:39.201421 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:39.202017 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 13:16:39.202236 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":524,"thread_start_us":322,"threads_started":1}
I20260619 13:16:39.204389 14012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:16:39 dist-test-slave-mkt3 krb5kdc[13784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 13:16:39 dist-test-slave-mkt3 krb5kdc[13784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 13:16:39 dist-test-slave-mkt3 krb5kdc[13784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:39.214908 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.198626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60560)
0619 13:16:39.198886 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:39.198905 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:39.198998 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:39.199717 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:39.199722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:39.200078 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:39.200256 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:39.200267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.202084 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.202089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:39.203150 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:39.203165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.205042 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.205048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:39.205208 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.205765 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:39.205781 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:39.208537 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:39.212185 (+ 3648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:39.212190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:39.212194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:39.212659 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:39.213191 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:39.213196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:39.213200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:39.213353 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:39.213972 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:39.213977 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:39.214145 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 13:16:39.214273 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 13:16:39.214528 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260619 13:16:39.215703 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.198789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.0.97.60:36787)
0619 13:16:39.199450 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:39.199455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:39.199475 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:16:39.199514 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:39.199519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:39.199582 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:39.199671 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:39.200423 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.201944 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.204209 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.204875 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.207198 (+ 2323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.208777 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:39.208783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:39.208788 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:39.208836 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:39.212063 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.212903 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.212908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.212911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.213059 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.213478 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.213483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.213486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.213809 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:39.215151 (+ 1342us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:39.215259 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 13:16:39.215447 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":371,"threads_started":1}
I20260619 13:16:39.218981 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.204995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39565 (local address 127.0.0.1:49472)
0619 13:16:39.205244 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:39.205260 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:39.205351 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:39.205785 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:39.205789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:39.206206 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:39.206369 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:39.206380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.207589 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.207595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:39.208949 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:39.208962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.210043 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.210049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:39.210279 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.210857 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:39.210878 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:39.213416 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:39.216700 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:39.216704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:39.216708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:39.217125 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:39.217498 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:39.217501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:39.217504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:39.217632 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:39.218185 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:39.218189 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:39.218312 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 13:16:39.218421 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 13:16:39.218661 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260619 13:16:39.219095 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.205091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.0.97.61:39565)
0619 13:16:39.205247 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:39.205253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:39.205306 (+ 53us) server_negotiation.cc:407] Connection header received
0619 13:16:39.205455 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:39.205460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:39.205528 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:39.205615 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:39.206525 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.207400 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.209109 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.209702 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.209794 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.213617 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:39.213623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:39.213626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:39.213657 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:39.216574 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.217224 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.217230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.217233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.217399 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.217742 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.217747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.217751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.218051 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:39.218186 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:39.218540 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 13:16:39.218746 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:16:39.219841 14012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:16:39.221937 13864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475: Bootstrap starting.
I20260619 13:16:39.222450 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.206259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.0.97.62:37141)
0619 13:16:39.206407 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:39.206415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:39.206480 (+ 65us) server_negotiation.cc:407] Connection header received
0619 13:16:39.206653 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:39.206657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:39.206728 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:39.206818 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:39.208044 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.208790 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.210115 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.210525 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.210642 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.215411 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:39.215417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:39.215421 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:39.215458 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:39.219424 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.220228 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.220237 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.220240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.220400 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.220838 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.220842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.220845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.221218 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:39.221485 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:39.221957 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 13:16:39.222160 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:16:39.226157 13864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475: Neither blocks nor log segments found. Creating new log.
I20260619 13:16:39.227432 13864 log.cc:824] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:16:39 dist-test-slave-mkt3 krb5kdc[13784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 13:16:39.230443 13864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475: No bootstrap required, opened a new log
Jun 19 13:16:39 dist-test-slave-mkt3 krb5kdc[13784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:16:39.233487 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.220714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.0.97.60:36787)
0619 13:16:39.220865 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:39.220870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:39.220884 (+ 14us) server_negotiation.cc:407] Connection header received
0619 13:16:39.220925 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:39.220929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:39.221002 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:39.221090 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:39.221902 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.222898 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.224415 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.224985 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.225094 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.228609 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:39.228615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:39.228620 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:39.228654 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:39.231452 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.232007 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.232012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.232016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.232127 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.232433 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.232438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.232441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.232678 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:39.232796 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:39.233119 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 13:16:39.233279 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:16:39.234282 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.220508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60564)
0619 13:16:39.220657 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:39.220674 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:39.220757 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:39.221148 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:39.221153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:39.221564 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:39.221731 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:39.221742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.223030 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.223035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:39.224244 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:39.224259 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.225244 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.225250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:39.225418 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.225986 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:39.226006 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:39.228441 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:39.231556 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:39.231561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:39.231564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:39.231909 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:39.232216 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:39.232221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:39.232225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:39.232346 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:39.232776 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:39.232780 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:39.232904 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 13:16:39.233693 (+ 789us) client_negotiation.cc:240] Negotiation successful
0619 13:16:39.233982 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 13:16:39.239500 14012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8: Bootstrap starting.
I20260619 13:16:39.240397 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.223782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.0.97.61:39565)
0619 13:16:39.223930 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:39.223934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:39.224019 (+ 85us) server_negotiation.cc:407] Connection header received
0619 13:16:39.224173 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:39.224179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:39.224239 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:39.224316 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:39.225404 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.226261 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.227898 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.228532 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.228613 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.234098 (+ 5485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:39.234104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:39.234109 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:39.234148 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:39.237615 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.238390 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.238394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.238397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.238555 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.238954 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.238959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.238962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.239299 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:39.239569 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:39.239868 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 13:16:39.240085 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:16:39.243108 14012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8: Neither blocks nor log segments found. Creating new log.
I20260619 13:16:39.244338 14012 log.cc:824] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:39.245400 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:16:39.246260 13864 raft_consensus.cc:348] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } }
I20260619 13:16:39.246639 13864 raft_consensus.cc:374] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:16:39.246745 13864 raft_consensus.cc:729] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d207bf922fea41e5a63b4093fb87b475, State: Initialized, Role: FOLLOWER
I20260619 13:16:39.247427 13864 consensus_queue.cc:260] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } }
I20260619 13:16:39.252064 14020 sys_catalog.cc:455] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } } }
I20260619 13:16:39.252429 14020 sys_catalog.cc:458] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:16:39.253468 14012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8: No bootstrap required, opened a new log
I20260619 13:16:39.254232 13864 sys_catalog.cc:564] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:16:39.264333 14012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } }
I20260619 13:16:39.264819 14012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:16:39.264946 14012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa254b53392447d9df7266bb25b43e8, State: Initialized, Role: FOLLOWER
I20260619 13:16:39.265662 14012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } }
I20260619 13:16:39.272529 14030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } } }
I20260619 13:16:39.272925 14030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:16:39.274816 14012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:16:39.285027 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.252501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60572 (local address 127.0.97.60:36787)
0619 13:16:39.252657 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:39.252663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:39.264466 (+ 11803us) server_negotiation.cc:407] Connection header received
0619 13:16:39.264720 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:39.264725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:39.264798 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:39.264870 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:39.267425 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.268587 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.270901 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.271638 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.271731 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.275481 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:39.275487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:39.275492 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:39.275527 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:39.279266 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.282477 (+ 3211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.282483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.282487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.282654 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.283112 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.283118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.283122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.283544 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:39.283826 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:39.284402 (+ 576us) server_negotiation.cc:299] Negotiation successful
0619 13:16:39.284642 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:16:39.290514 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.258730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.0.97.61:39565)
0619 13:16:39.258892 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:39.258898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:39.268057 (+ 9159us) server_negotiation.cc:407] Connection header received
0619 13:16:39.268506 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:39.268513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:39.268601 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:39.268673 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:39.276154 (+ 7481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.277319 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.279301 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.279980 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.280075 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.283604 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:39.283611 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:39.283617 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:39.283661 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:39.287176 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.288015 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.288021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.288024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.288191 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.288668 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.288674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.288678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.289036 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:39.289277 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:39.289830 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 13:16:39.290117 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:16:39.295128 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.252554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.0.97.62:37141)
0619 13:16:39.252708 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:39.252714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:39.262309 (+ 9595us) server_negotiation.cc:407] Connection header received
0619 13:16:39.263273 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:39.263279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:39.263374 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:39.263449 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:39.271113 (+ 7664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.272315 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.274493 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.275212 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.275315 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.287776 (+ 12461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:39.287784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:39.287789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:39.287836 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:39.291546 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.292677 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.292683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.292687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.292872 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.293334 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.293339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.293343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.293684 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:39.293955 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:39.294497 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 13:16:39.294761 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 13:16:39.304780 14044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:16:39.304953 14044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:16:39.311350 13945 master_runner.cc:186] Error getting master registration for 127.0.97.61:39565: OK, instance_id { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" instance_seqno: 1781874998704495 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:16:39.321408 13945 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:16:39.304759 13797 master_runner.cc:186] Error getting master registration for 127.0.97.61:39565: OK, instance_id { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" instance_seqno: 1781874998704495 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:16:39.321723 13797 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 13:16:39.323961 14051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:16:39.324087 14051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 13:16:39.406281 13966 tablet.cc:2369] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:16:39.423420 13938 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } attempt: 5
Jun 19 13:16:39 dist-test-slave-mkt3 krb5kdc[13784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781874998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 13:16:39.440719 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.424001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36787 (local address 127.0.0.1:60582)
0619 13:16:39.424300 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:39.424321 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:39.424453 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:39.424962 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:39.424968 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:39.425482 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:39.425703 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:39.425717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.427202 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.427208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:39.428640 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:39.428656 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.429777 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.429784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:39.430018 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.430714 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:39.430739 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:39.433674 (+ 2935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:39.437961 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:39.437966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:39.437971 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:39.438496 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:39.438912 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:39.438917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:39.438921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:39.439087 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:39.439699 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:39.439705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:39.439868 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 13:16:39.440029 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 13:16:39.440317 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260619 13:16:39.440725 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:39.424085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.0.97.60:36787)
0619 13:16:39.424294 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:39.424300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:39.424405 (+ 105us) server_negotiation.cc:407] Connection header received
0619 13:16:39.424586 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:39.424599 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:39.424676 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:39.424765 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:39.425897 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.427036 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.428791 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:39.429591 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:39.429718 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:39.433910 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:39.433916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:39.433921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:39.433962 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:39.437804 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.438629 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.438635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.438639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.438795 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:39.439200 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:39.439206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:39.439209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:39.439570 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:39.439736 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:39.440143 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 13:16:39.440370 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:16:39.446897 13938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad: Bootstrap starting.
I20260619 13:16:39.450925 13938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad: Neither blocks nor log segments found. Creating new log.
I20260619 13:16:39.453372 13938 log.cc:824] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:39.456664 13938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad: No bootstrap required, opened a new log
I20260619 13:16:39.467154 13938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } }
I20260619 13:16:39.467531 13938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:16:39.467646 13938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900594647ab54e26b7dd5d1288b14bad, State: Initialized, Role: FOLLOWER
I20260619 13:16:39.468268 13938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } }
W20260619 13:16:39.471652 13818 tablet.cc:2369] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:16:39.473027 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } } }
I20260619 13:16:39.473595 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:16:39.474396 13938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:16:39.494530 13871 master_runner.cc:263] No leader master found from master 900594647ab54e26b7dd5d1288b14bad
I20260619 13:16:39.494679 13871 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 13:16:39.500728 14065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:16:39.500849 14065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:16:39.528201 14020 raft_consensus.cc:482] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:16:39.528517 14020 raft_consensus.cc:504] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } }
I20260619 13:16:39.530257 14020 leader_election.cc:288] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 900594647ab54e26b7dd5d1288b14bad (127.0.97.61:39565), 2fa254b53392447d9df7266bb25b43e8 (127.0.97.60:36787)
I20260619 13:16:39.531387 13913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d207bf922fea41e5a63b4093fb87b475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900594647ab54e26b7dd5d1288b14bad" is_pre_election: true
I20260619 13:16:39.532015 13913 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d207bf922fea41e5a63b4093fb87b475 in term 0.
I20260619 13:16:39.533073 13987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d207bf922fea41e5a63b4093fb87b475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa254b53392447d9df7266bb25b43e8" is_pre_election: true
I20260619 13:16:39.533659 13987 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d207bf922fea41e5a63b4093fb87b475 in term 0.
I20260619 13:16:39.534193 13816 leader_election.cc:302] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900594647ab54e26b7dd5d1288b14bad, d207bf922fea41e5a63b4093fb87b475; no voters:
I20260619 13:16:39.539116 14020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:16:39.539265 14020 raft_consensus.cc:482] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:16:39.539366 14020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:16:39.542616 14020 raft_consensus.cc:504] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } }
I20260619 13:16:39.543665 13913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d207bf922fea41e5a63b4093fb87b475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900594647ab54e26b7dd5d1288b14bad"
I20260619 13:16:39.543879 13913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:16:39.547155 13913 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d207bf922fea41e5a63b4093fb87b475 in term 1.
I20260619 13:16:39.547612 13816 leader_election.cc:302] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 900594647ab54e26b7dd5d1288b14bad, d207bf922fea41e5a63b4093fb87b475; no voters:
I20260619 13:16:39.550908 13987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d207bf922fea41e5a63b4093fb87b475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa254b53392447d9df7266bb25b43e8"
I20260619 13:16:39.551131 13987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:16:39.554452 13987 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d207bf922fea41e5a63b4093fb87b475 in term 1.
I20260619 13:16:39.555115 14020 leader_election.cc:288] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [CANDIDATE]: Term 1 election: Requested vote from peers 900594647ab54e26b7dd5d1288b14bad (127.0.97.61:39565), 2fa254b53392447d9df7266bb25b43e8 (127.0.97.60:36787)
I20260619 13:16:39.555331 14020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:16:39.555475 14020 raft_consensus.cc:686] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [term 1 LEADER]: Becoming Leader. State: Replica: d207bf922fea41e5a63b4093fb87b475, State: Running, Role: LEADER
I20260619 13:16:39.556035 14020 consensus_queue.cc:237] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } }
I20260619 13:16:39.577687 14067 sys_catalog.cc:455] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d207bf922fea41e5a63b4093fb87b475. Latest consensus state: current_term: 1 leader_uuid: "d207bf922fea41e5a63b4093fb87b475" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } } }
I20260619 13:16:39.578125 14067 sys_catalog.cc:458] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [sys.catalog]: This master's current role is: LEADER
I20260619 13:16:39.585440 14069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:16:39.587998 14069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:16:39.608310 13987 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8 [term 1 FOLLOWER]: Refusing update from remote peer d207bf922fea41e5a63b4093fb87b475: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:16:39.609319 14067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:16:39.616770 13913 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad [term 1 FOLLOWER]: Refusing update from remote peer d207bf922fea41e5a63b4093fb87b475: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:16:39.617635 14067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [LEADER]: Connected to new peer: Peer: permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:16:39.656280 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad [sys.catalog]: SysCatalogTable state changed. Reason: New leader d207bf922fea41e5a63b4093fb87b475. Latest consensus state: current_term: 1 leader_uuid: "d207bf922fea41e5a63b4093fb87b475" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } } }
I20260619 13:16:39.656662 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:16:39.661027 14070 mvcc.cc:205] Tried to move back new op lower bound from 7298559998383616000 to 7298559998195310592. Current Snapshot: MvccSnapshot[applied={T|T < 7298559998383616000}]
I20260619 13:16:39.664724 14072 mvcc.cc:205] Tried to move back new op lower bound from 7298559998383616000 to 7298559998195310592. Current Snapshot: MvccSnapshot[applied={T|T < 7298559998383616000}]
I20260619 13:16:39.665361 14020 sys_catalog.cc:455] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d207bf922fea41e5a63b4093fb87b475" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } } }
I20260619 13:16:39.665704 14020 sys_catalog.cc:458] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [sys.catalog]: This master's current role is: LEADER
I20260619 13:16:39.667995 14069 catalog_manager.cc:1347] Generated new cluster ID: e3cf76ff7b954322af6908c45b8a8cab
I20260619 13:16:39.668102 14069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:16:39.683066 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d207bf922fea41e5a63b4093fb87b475" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } } }
I20260619 13:16:39.683362 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:16:39.685135 14030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d207bf922fea41e5a63b4093fb87b475. Latest consensus state: current_term: 1 leader_uuid: "d207bf922fea41e5a63b4093fb87b475" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } } }
I20260619 13:16:39.685464 14030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:16:39.687582 14067 sys_catalog.cc:455] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d207bf922fea41e5a63b4093fb87b475" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } } }
I20260619 13:16:39.687850 14067 sys_catalog.cc:458] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475 [sys.catalog]: This master's current role is: LEADER
I20260619 13:16:39.697937 14030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d207bf922fea41e5a63b4093fb87b475" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d207bf922fea41e5a63b4093fb87b475" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37141 } } peers { permanent_uuid: "900594647ab54e26b7dd5d1288b14bad" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39565 } } peers { permanent_uuid: "2fa254b53392447d9df7266bb25b43e8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36787 } } }
I20260619 13:16:39.698169 14030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:16:39.718632 14069 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:16:39.720408 14069 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:16:39.736307 14069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d207bf922fea41e5a63b4093fb87b475: Generated new TSK 0
I20260619 13:16:39.737327 14069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:16:40.325953 14051 catalog_manager.cc:1259] Loaded cluster ID: e3cf76ff7b954322af6908c45b8a8cab
I20260619 13:16:40.326054 14051 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8: loading cluster ID for follower catalog manager: success
I20260619 13:16:40.328368 14051 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8: acquiring CA information for follower catalog manager: success
I20260619 13:16:40.330159 14051 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2fa254b53392447d9df7266bb25b43e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:16:40.502732 14065 catalog_manager.cc:1259] Loaded cluster ID: e3cf76ff7b954322af6908c45b8a8cab
I20260619 13:16:40.502823 14065 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad: loading cluster ID for follower catalog manager: success
I20260619 13:16:40.505120 14065 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad: acquiring CA information for follower catalog manager: success
I20260619 13:16:40.506826 14065 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 900594647ab54e26b7dd5d1288b14bad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:16:40.812155 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 13797
I20260619 13:16:40.813315 13859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:40.924003 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 13797
W20260619 13:16:40.940320 13964 connection.cc:535] server connection from 127.0.0.1:60560 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 13:16:40.940662 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 13871
I20260619 13:16:40.941671 13933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:41.046928 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 13871
I20260619 13:16:41.062879 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 13945
I20260619 13:16:41.063900 14007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:41.167436 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 13945
2026-06-19T13:16:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4519 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:16:41 dist-test-slave-mkt3 krb5kdc[14086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:16:41 dist-test-slave-mkt3 krb5kdc[14086](info): set up 2 sockets
Jun 19 13:16:41 dist-test-slave-mkt3 krb5kdc[14086](info): commencing operation
krb5kdc: starting...
W20260619 13:16:42.166105 388 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:16:42 dist-test-slave-mkt3 krb5kdc[14086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:16:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:16:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:16:42.366889 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33497
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:16:42.678241 14099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:42.678508 14099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:42.678635 14099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:42.685577 14099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:16:42.685668 14099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:42.685735 14099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:42.685814 14099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:16:42.685884 14099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:16:42.695176 14099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:42.696350 14099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:42.697594 14099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:42.708380 14107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:42.708410 14104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:42.708334 14105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:42.709143 14099 server_base.cc:1034] running on GCE node
I20260619 13:16:42.709905 14099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:42.711187 14099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:42.712399 14099 hybrid_clock.cc:648] HybridClock initialized: now 1781875002712343 us; error 40 us; skew 500 ppm
I20260619 13:16:42.712723 14099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:42 dist-test-slave-mkt3 krb5kdc[14086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:42.715595 14099 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:16:42.717003 14099 webserver.cc:460] Webserver started at http://127.0.97.62:43263/ using document root <none> and password file <none>
I20260619 13:16:42.717654 14099 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:42.717763 14099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:42.718072 14099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:16:42.720940 14099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "8c03c838f3cf4031b3ce25b0828452ae"
format_stamp: "Formatted at 2026-06-19 13:16:42 on dist-test-slave-mkt3"
server_key: "6b3aab3e6cd8fad0a67d77e0b9de71b1"
server_key_iv: "72722c29e4408304f36e496e4f7dea52"
server_key_version: "encryptionkey@0"
I20260619 13:16:42.721673 14099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "8c03c838f3cf4031b3ce25b0828452ae"
format_stamp: "Formatted at 2026-06-19 13:16:42 on dist-test-slave-mkt3"
server_key: "6b3aab3e6cd8fad0a67d77e0b9de71b1"
server_key_iv: "72722c29e4408304f36e496e4f7dea52"
server_key_version: "encryptionkey@0"
I20260619 13:16:42.726593 14099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:16:42.729775 14114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:42.731037 14099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:16:42.731196 14099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "8c03c838f3cf4031b3ce25b0828452ae"
format_stamp: "Formatted at 2026-06-19 13:16:42 on dist-test-slave-mkt3"
server_key: "6b3aab3e6cd8fad0a67d77e0b9de71b1"
server_key_iv: "72722c29e4408304f36e496e4f7dea52"
server_key_version: "encryptionkey@0"
I20260619 13:16:42.731374 14099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:42.758441 14099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:42.762012 14099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:42.762249 14099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:42.778287 14099 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33497
I20260619 13:16:42.778329 14165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33497 every 8 connection(s)
I20260619 13:16:42.779558 14099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:16:42.783208 14166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:16:42.785476 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 14099
I20260619 13:16:42.785646 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:16:42.786334 388 external_mini_cluster.cc:1428] Setting key 4110811446f2d0fa8c575dca93f45b9b
I20260619 13:16:42.794632 14166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae: Bootstrap starting.
I20260619 13:16:42.798313 14166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae: Neither blocks nor log segments found. Creating new log.
I20260619 13:16:42.799580 14166 log.cc:824] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:16:42 dist-test-slave-mkt3 krb5kdc[14086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:42.802541 14166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae: No bootstrap required, opened a new log
I20260619 13:16:42.808073 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:42.788095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.0.97.62:33497)
0619 13:16:42.788804 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:42.788817 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:42.788858 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:16:42.790538 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:42.790561 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:42.790939 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:42.791261 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:42.792987 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:42.794324 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:42.796094 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:42.796752 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:42.796870 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:42.801462 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:42.801470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:42.801483 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:42.801516 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:42.804344 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:42.804978 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:42.804987 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:42.804992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:42.805149 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:42.805518 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:42.805522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:42.805524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:42.806041 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:42.806326 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:42.806732 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 13:16:42.807117 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":305,"threads_started":1}
I20260619 13:16:42.810210 14166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c03c838f3cf4031b3ce25b0828452ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33497 } }
I20260619 13:16:42.810564 14166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:16:42.810655 14166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c03c838f3cf4031b3ce25b0828452ae, State: Initialized, Role: FOLLOWER
I20260619 13:16:42.811180 14166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c03c838f3cf4031b3ce25b0828452ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33497 } }
I20260619 13:16:42.811405 14166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:16:42.811539 14166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:16:42.811599 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:16:42.811681 14166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:16:42.813807 14166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c03c838f3cf4031b3ce25b0828452ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33497 } }
I20260619 13:16:42.814322 14166 leader_election.cc:302] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c03c838f3cf4031b3ce25b0828452ae; no voters:
I20260619 13:16:42.814853 14166 leader_election.cc:288] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:16:42.814951 14171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:16:42.815555 14171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [term 1 LEADER]: Becoming Leader. State: Replica: 8c03c838f3cf4031b3ce25b0828452ae, State: Running, Role: LEADER
I20260619 13:16:42.816062 14171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c03c838f3cf4031b3ce25b0828452ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33497 } }
I20260619 13:16:42.816686 14166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:16:42.818521 14172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c03c838f3cf4031b3ce25b0828452ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c03c838f3cf4031b3ce25b0828452ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33497 } } }
I20260619 13:16:42.818900 14172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [sys.catalog]: This master's current role is: LEADER
I20260619 13:16:42.820195 14177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:16:42.820160 14173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c03c838f3cf4031b3ce25b0828452ae. Latest consensus state: current_term: 1 leader_uuid: "8c03c838f3cf4031b3ce25b0828452ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c03c838f3cf4031b3ce25b0828452ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33497 } } }
I20260619 13:16:42.820479 14173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae [sys.catalog]: This master's current role is: LEADER
I20260619 13:16:42.824935 14177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:16:42.832358 14177 catalog_manager.cc:1347] Generated new cluster ID: 9d6f0c9c57f444d6b73b956a8fa65d60
I20260619 13:16:42.832466 14177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:16:42.853149 14177 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:16:42.854418 14177 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:16:42.861277 14177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c03c838f3cf4031b3ce25b0828452ae: Generated new TSK 0
I20260619 13:16:42.862052 14177 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:16:42.871024 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 14099
I20260619 13:16:42.872049 14161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:42.962200 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 14099
2026-06-19T13:16:42Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1790 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:16:43 dist-test-slave-mkt3 krb5kdc[14194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:16:43 dist-test-slave-mkt3 krb5kdc[14194](info): set up 2 sockets
Jun 19 13:16:43 dist-test-slave-mkt3 krb5kdc[14194](info): commencing operation
krb5kdc: starting...
W20260619 13:16:43.954136 388 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:16:44 dist-test-slave-mkt3 krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:16:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:16:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:16:44.159490 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38653
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:16:44.465555 14207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:44.465816 14207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:44.465947 14207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:44.472846 14207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:16:44.472934 14207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:44.472986 14207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:44.473042 14207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:16:44.473083 14207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:16:44.482568 14207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:44.483794 14207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:44.485006 14207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:44.495700 14212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:44.495718 14213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:44.495913 14215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:44.496492 14207 server_base.cc:1034] running on GCE node
I20260619 13:16:44.497133 14207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:44.498454 14207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:44.499665 14207 hybrid_clock.cc:648] HybridClock initialized: now 1781875004499622 us; error 27 us; skew 500 ppm
I20260619 13:16:44.500013 14207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:44 dist-test-slave-mkt3 krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:44.510752 14207 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:16:44.512197 14207 webserver.cc:460] Webserver started at http://127.0.97.62:41293/ using document root <none> and password file <none>
I20260619 13:16:44.512863 14207 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:44.512979 14207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:44.513253 14207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:16:44.516144 14207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "e3cebe8a96294ffcbd37162dcfd86876"
format_stamp: "Formatted at 2026-06-19 13:16:44 on dist-test-slave-mkt3"
server_key: "9f5eab2a4d547e1e9918724b40ccc111"
server_key_iv: "96e9e69e8486bb4e3200c9fdd1d17f16"
server_key_version: "encryptionkey@0"
I20260619 13:16:44.516860 14207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "e3cebe8a96294ffcbd37162dcfd86876"
format_stamp: "Formatted at 2026-06-19 13:16:44 on dist-test-slave-mkt3"
server_key: "9f5eab2a4d547e1e9918724b40ccc111"
server_key_iv: "96e9e69e8486bb4e3200c9fdd1d17f16"
server_key_version: "encryptionkey@0"
I20260619 13:16:44.521711 14207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:16:44.524969 14222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:44.526250 14207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:16:44.526403 14207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "e3cebe8a96294ffcbd37162dcfd86876"
format_stamp: "Formatted at 2026-06-19 13:16:44 on dist-test-slave-mkt3"
server_key: "9f5eab2a4d547e1e9918724b40ccc111"
server_key_iv: "96e9e69e8486bb4e3200c9fdd1d17f16"
server_key_version: "encryptionkey@0"
I20260619 13:16:44.526567 14207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:44.550590 14207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:44.554318 14207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:44.554562 14207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:44.570808 14207 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38653
I20260619 13:16:44.570844 14273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38653 every 8 connection(s)
I20260619 13:16:44.571997 14207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:16:44.575385 14274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:16:44.578421 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 14207
I20260619 13:16:44.578624 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:16:44.579406 388 external_mini_cluster.cc:1428] Setting key b5748100677e5434b33258616ae6eb3b
I20260619 13:16:44.587443 14274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: Bootstrap starting.
I20260619 13:16:44.590886 14274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: Neither blocks nor log segments found. Creating new log.
I20260619 13:16:44.592108 14274 log.cc:824] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:16:44 dist-test-slave-mkt3 krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:44.594944 14274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: No bootstrap required, opened a new log
I20260619 13:16:44.601585 14277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:44.581391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51270 (local address 127.0.97.62:38653)
0619 13:16:44.582092 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:44.582104 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:44.582152 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:16:44.583708 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:44.583729 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:44.584106 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:44.584422 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:44.586056 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:44.587315 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:44.589052 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:44.589700 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:44.589826 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:44.594480 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:44.594488 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:44.594503 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:44.594539 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:44.597578 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:44.598344 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:44.598351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:44.598357 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:44.598517 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:44.598887 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:44.598891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:44.598893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:44.599412 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:44.599698 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:44.600168 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 13:16:44.600560 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":307,"threads_started":1}
I20260619 13:16:44.602663 14274 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } }
I20260619 13:16:44.603004 14274 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:16:44.603103 14274 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cebe8a96294ffcbd37162dcfd86876, State: Initialized, Role: FOLLOWER
I20260619 13:16:44.603681 14274 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } }
I20260619 13:16:44.603917 14274 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:16:44.604053 14274 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:16:44.604238 14274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:16:44.606408 14274 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } }
I20260619 13:16:44.606848 14274 leader_election.cc:302] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3cebe8a96294ffcbd37162dcfd86876; no voters:
I20260619 13:16:44.607342 14274 leader_election.cc:288] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:16:44.607476 14279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:16:44.608067 14279 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 1 LEADER]: Becoming Leader. State: Replica: e3cebe8a96294ffcbd37162dcfd86876, State: Running, Role: LEADER
I20260619 13:16:44.608603 14279 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } }
I20260619 13:16:44.609285 14274 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:16:44.611274 14281 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3cebe8a96294ffcbd37162dcfd86876. Latest consensus state: current_term: 1 leader_uuid: "e3cebe8a96294ffcbd37162dcfd86876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } } }
I20260619 13:16:44.611598 14281 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [sys.catalog]: This master's current role is: LEADER
I20260619 13:16:44.612916 14280 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3cebe8a96294ffcbd37162dcfd86876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } } }
I20260619 13:16:44.613201 14280 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [sys.catalog]: This master's current role is: LEADER
I20260619 13:16:44.613209 14284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:16:44.619208 14284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:16:44.628552 14284 catalog_manager.cc:1347] Generated new cluster ID: 57362b2b1f3344fca7ea091f29005da8
I20260619 13:16:44.628667 14284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:16:44.638772 14284 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:16:44.639974 14284 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:16:44.648813 14284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: Generated new TSK 0
I20260619 13:16:44.649591 14284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:16:44.708642 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38653
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:16:45.019838 14300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:45.020092 14300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:45.020215 14300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:45.027364 14300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:45.027457 14300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:45.027593 14300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:16:45.037106 14300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:45.038316 14300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:45.039564 14300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:45.051313 14305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:45.051355 14308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:45.051347 14306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:45.052069 14300 server_base.cc:1034] running on GCE node
I20260619 13:16:45.052634 14300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:45.053452 14300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:45.054629 14300 hybrid_clock.cc:648] HybridClock initialized: now 1781875005054584 us; error 37 us; skew 500 ppm
I20260619 13:16:45.054914 14300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:45 dist-test-slave-mkt3 krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:45.057824 14300 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:16:45.059173 14300 webserver.cc:460] Webserver started at http://127.0.97.1:36521/ using document root <none> and password file <none>
I20260619 13:16:45.059788 14300 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:45.059897 14300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:45.060159 14300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:16:45.062897 14300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415"
format_stamp: "Formatted at 2026-06-19 13:16:45 on dist-test-slave-mkt3"
server_key: "426a65bee064dc6a8128e69f889afc62"
server_key_iv: "f7489ac7916a92fb0a908b887bd482be"
server_key_version: "encryptionkey@0"
I20260619 13:16:45.063598 14300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415"
format_stamp: "Formatted at 2026-06-19 13:16:45 on dist-test-slave-mkt3"
server_key: "426a65bee064dc6a8128e69f889afc62"
server_key_iv: "f7489ac7916a92fb0a908b887bd482be"
server_key_version: "encryptionkey@0"
I20260619 13:16:45.068279 14300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:16:45.071447 14315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:45.072639 14300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 13:16:45.072778 14300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415"
format_stamp: "Formatted at 2026-06-19 13:16:45 on dist-test-slave-mkt3"
server_key: "426a65bee064dc6a8128e69f889afc62"
server_key_iv: "f7489ac7916a92fb0a908b887bd482be"
server_key_version: "encryptionkey@0"
I20260619 13:16:45.072942 14300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:45.094964 14300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:45.099038 14300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:45.099288 14300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:45.100179 14300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:16:45.101440 14300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:16:45.101517 14300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:45.101625 14300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:16:45.101683 14300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:45.130631 14300 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42847
I20260619 13:16:45.130707 14427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42847 every 8 connection(s)
I20260619 13:16:45.131889 14300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:16:45.138820 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 14300
I20260619 13:16:45.138988 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:16:45.139597 388 external_mini_cluster.cc:1428] Setting key 68404f94ca4ef640ab02ccb5a2b0d648
Jun 19 13:16:45 dist-test-slave-mkt3 krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:45.151227 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:45.134285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42439 (local address 127.0.97.62:38653)
0619 13:16:45.134730 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:45.134735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:45.135695 (+ 960us) server_negotiation.cc:407] Connection header received
0619 13:16:45.137475 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:45.137481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:45.137564 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:45.137664 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:45.139202 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:45.139984 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:45.141762 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:45.142222 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:45.142332 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:45.145588 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:45.145592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:45.145596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:45.145628 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:45.148083 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:45.148783 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:45.148789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:45.148793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:45.148954 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:45.149462 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:45.149466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:45.149468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:45.149764 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:45.149931 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:45.150731 (+ 800us) server_negotiation.cc:299] Negotiation successful
0619 13:16:45.150969 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":201,"threads_started":1}
I20260619 13:16:45.152370 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:45.134837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:42439)
0619 13:16:45.135527 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:45.135569 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:45.137201 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:45.137941 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:45.137949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:45.138446 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:45.139017 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:45.139041 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:45.140136 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:45.140140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:45.141631 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:45.141642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:45.142443 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:45.142451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:45.142670 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:45.143294 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:45.143315 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:45.145373 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:45.148206 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:45.148215 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:45.148231 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:45.148671 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:45.149088 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:45.149094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:45.149098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:45.149332 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:45.149904 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:45.149912 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:45.150290 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 13:16:45.150599 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 13:16:45.151092 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":258,"threads_started":1}
I20260619 13:16:45.153900 14428 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38653
I20260619 13:16:45.154188 14428 heartbeater.cc:464] Registering TS with master...
I20260619 13:16:45.154875 14428 heartbeater.cc:511] Master 127.0.97.62:38653 requested a full tablet report, sending...
I20260619 13:16:45.156958 14239 ts_manager.cc:194] Registered new tserver with Master: d6f26ac0e3bd4f7b94f37b5b0317f415 (127.0.97.1:42847)
I20260619 13:16:45.158438 14239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42439
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:16:45.184527 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38653
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:16:45.491325 14434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:45.491590 14434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:45.491688 14434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:45.498605 14434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:45.498700 14434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:45.498821 14434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:16:45.508085 14434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:45.509312 14434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:45.510666 14434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:45.522651 14439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:45.522616 14442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:45.522708 14440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:45.523255 14434 server_base.cc:1034] running on GCE node
I20260619 13:16:45.523792 14434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:45.524662 14434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:45.525882 14434 hybrid_clock.cc:648] HybridClock initialized: now 1781875005525811 us; error 40 us; skew 500 ppm
I20260619 13:16:45.526186 14434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:45 dist-test-slave-mkt3 krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:45.528992 14434 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:16:45.530493 14434 webserver.cc:460] Webserver started at http://127.0.97.2:42365/ using document root <none> and password file <none>
I20260619 13:16:45.531134 14434 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:45.531246 14434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:45.531518 14434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:16:45.534301 14434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "1dfb376a3034440384ab83181d050d9c"
format_stamp: "Formatted at 2026-06-19 13:16:45 on dist-test-slave-mkt3"
server_key: "e3edc0d271864ff566c586e5ec969e0e"
server_key_iv: "0fb1ad2900922fbf52a0167e991bc31c"
server_key_version: "encryptionkey@0"
I20260619 13:16:45.535043 14434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1dfb376a3034440384ab83181d050d9c"
format_stamp: "Formatted at 2026-06-19 13:16:45 on dist-test-slave-mkt3"
server_key: "e3edc0d271864ff566c586e5ec969e0e"
server_key_iv: "0fb1ad2900922fbf52a0167e991bc31c"
server_key_version: "encryptionkey@0"
I20260619 13:16:45.539778 14434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:16:45.542963 14449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:45.544144 14434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260619 13:16:45.544287 14434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "1dfb376a3034440384ab83181d050d9c"
format_stamp: "Formatted at 2026-06-19 13:16:45 on dist-test-slave-mkt3"
server_key: "e3edc0d271864ff566c586e5ec969e0e"
server_key_iv: "0fb1ad2900922fbf52a0167e991bc31c"
server_key_version: "encryptionkey@0"
I20260619 13:16:45.544458 14434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:45.562286 14434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:45.565735 14434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:45.566012 14434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:45.566859 14434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:16:45.568076 14434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:16:45.568148 14434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:45.568254 14434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:16:45.568303 14434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:45.606449 14434 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45285
I20260619 13:16:45.606480 14561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45285 every 8 connection(s)
I20260619 13:16:45.607654 14434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:16:45.613229 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 14434
I20260619 13:16:45.613384 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:16:45.613795 388 external_mini_cluster.cc:1428] Setting key c9c7eaf85bac65df4cefaccfc6bcb424
Jun 19 13:16:45 dist-test-slave-mkt3 krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:45.626470 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:45.610035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57015 (local address 127.0.97.62:38653)
0619 13:16:45.610203 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:45.610207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:45.611425 (+ 1218us) server_negotiation.cc:407] Connection header received
0619 13:16:45.613177 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:45.613183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:45.613265 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:45.613353 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:45.615014 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:45.615937 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:45.617429 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:45.617858 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:45.617971 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:45.621118 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:45.621123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:45.621126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:45.621157 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:45.623558 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:45.624370 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:45.624373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:45.624375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:45.624500 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:45.624898 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:45.624901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:45.624903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:45.625141 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:45.625242 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:45.625993 (+ 751us) server_negotiation.cc:299] Negotiation successful
0619 13:16:45.626207 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 13:16:45.627703 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:45.610595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:57015)
0619 13:16:45.611266 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:45.611303 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:45.612925 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:45.613586 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:45.613594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:45.614181 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:45.614828 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:45.614846 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:45.616079 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:45.616085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:45.617289 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:45.617301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:45.618065 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:45.618071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:45.618281 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:45.618873 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:45.618899 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:45.620907 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:45.623701 (+ 2794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:45.623709 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:45.623726 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:45.624266 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:45.624590 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:45.624595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:45.624598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:45.624816 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:45.625248 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:45.625256 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:45.625624 (+ 368us) client_negotiation.cc:769] Sending connection context
0619 13:16:45.625945 (+ 321us) client_negotiation.cc:240] Negotiation successful
0619 13:16:45.626446 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":259,"threads_started":1}
I20260619 13:16:45.629084 14562 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38653
I20260619 13:16:45.629369 14562 heartbeater.cc:464] Registering TS with master...
I20260619 13:16:45.630098 14562 heartbeater.cc:511] Master 127.0.97.62:38653 requested a full tablet report, sending...
I20260619 13:16:45.631484 14239 ts_manager.cc:194] Registered new tserver with Master: 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285)
I20260619 13:16:45.632243 14239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57015
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:16:45.658437 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38653
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:16:45.964145 14567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:45.964413 14567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:45.964540 14567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:45.971976 14567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:45.972090 14567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:45.972244 14567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:16:45.982147 14567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:45.983458 14567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:45.984741 14567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:45.997148 14572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:45.997188 14573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:45.997113 14575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:46.003228 14567 server_base.cc:1034] running on GCE node
I20260619 13:16:46.003892 14567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:46.004905 14567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:46.006093 14567 hybrid_clock.cc:648] HybridClock initialized: now 1781875006006032 us; error 59 us; skew 500 ppm
I20260619 13:16:46.006423 14567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:46 dist-test-slave-mkt3 krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:46.009523 14567 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:16:46.011152 14567 webserver.cc:460] Webserver started at http://127.0.97.3:34633/ using document root <none> and password file <none>
I20260619 13:16:46.011857 14567 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:46.011974 14567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:46.012249 14567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:16:46.015298 14567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "3853ee3fe7ba42d7be2b0686ff189e24"
format_stamp: "Formatted at 2026-06-19 13:16:46 on dist-test-slave-mkt3"
server_key: "370e28d5d3d969cebc320e10449b0c71"
server_key_iv: "468b3615154ccefadde62f4f5062a24f"
server_key_version: "encryptionkey@0"
I20260619 13:16:46.016083 14567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3853ee3fe7ba42d7be2b0686ff189e24"
format_stamp: "Formatted at 2026-06-19 13:16:46 on dist-test-slave-mkt3"
server_key: "370e28d5d3d969cebc320e10449b0c71"
server_key_iv: "468b3615154ccefadde62f4f5062a24f"
server_key_version: "encryptionkey@0"
I20260619 13:16:46.021148 14567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:16:46.024533 14582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:46.025902 14567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:16:46.026055 14567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "3853ee3fe7ba42d7be2b0686ff189e24"
format_stamp: "Formatted at 2026-06-19 13:16:46 on dist-test-slave-mkt3"
server_key: "370e28d5d3d969cebc320e10449b0c71"
server_key_iv: "468b3615154ccefadde62f4f5062a24f"
server_key_version: "encryptionkey@0"
I20260619 13:16:46.026234 14567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:46.042797 14567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:46.046602 14567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:46.046864 14567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:46.047832 14567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:16:46.049170 14567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:16:46.049240 14567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:46.049333 14567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:16:46.049377 14567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:46.078578 14567 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44113
I20260619 13:16:46.078612 14694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44113 every 8 connection(s)
I20260619 13:16:46.079809 14567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:16:46.087159 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 14567
I20260619 13:16:46.087344 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:16:46.087903 388 external_mini_cluster.cc:1428] Setting key 1d2402fff9f343e49618243a6eb1265b
Jun 19 13:16:46 dist-test-slave-mkt3 krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:46.098987 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.082296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50241 (local address 127.0.97.62:38653)
0619 13:16:46.082473 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:46.082477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:46.083731 (+ 1254us) server_negotiation.cc:407] Connection header received
0619 13:16:46.085591 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:46.085597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:46.085688 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:46.085777 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:46.087402 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.088450 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.090237 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.090723 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.090841 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.094182 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:46.094187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:46.094191 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:46.094224 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:46.096482 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:46.097126 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:46.097130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:46.097132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:46.097237 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:46.097554 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:46.097556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:46.097558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:46.097800 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:46.097938 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:46.098583 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 13:16:46.098740 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:16:46.099900 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.082820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:50241)
0619 13:16:46.083553 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:46.083591 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:46.085324 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:46.086058 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:46.086066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:46.086580 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:46.087194 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:46.087213 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.088626 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.088631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:46.090100 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:46.090113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.090970 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.090977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:46.091190 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.091771 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:46.091797 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:46.094000 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:46.096606 (+ 2606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:46.096612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:46.096625 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:46.097038 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:46.097309 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:46.097312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:46.097314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:46.097477 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:46.097960 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:46.097966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:46.098287 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 13:16:46.098513 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 13:16:46.098893 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":308,"threads_started":1}
I20260619 13:16:46.101179 14695 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38653
I20260619 13:16:46.101488 14695 heartbeater.cc:464] Registering TS with master...
I20260619 13:16:46.102300 14695 heartbeater.cc:511] Master 127.0.97.62:38653 requested a full tablet report, sending...
I20260619 13:16:46.103682 14239 ts_manager.cc:194] Registered new tserver with Master: 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3:44113)
I20260619 13:16:46.104409 14239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50241
I20260619 13:16:46.114974 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:16:46.131856 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.119513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51280 (local address 127.0.97.62:38653)
0619 13:16:46.119704 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:46.119711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:46.120109 (+ 398us) server_negotiation.cc:407] Connection header received
0619 13:16:46.120248 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:46.120252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:46.120317 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:46.120401 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:46.121705 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.122476 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.123780 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.124169 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.124259 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.126546 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:46.126551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:46.126555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:46.126591 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:46.129045 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:46.129771 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:46.129792 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:46.129795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:46.129992 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:46.130455 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:46.130459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:46.130461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:46.130864 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:46.131094 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:46.131446 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 13:16:46.131621 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:16:46.135886 14239 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:16:46.142557 14239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:16:46.161458 14428 heartbeater.cc:503] Master 127.0.97.62:38653 was elected leader, sending a full tablet report...
I20260619 13:16:46.175220 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.154229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39122 (local address 127.0.97.1:42847)
0619 13:16:46.166011 (+ 11782us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:46.166021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:46.166769 (+ 748us) server_negotiation.cc:407] Connection header received
0619 13:16:46.166849 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:46.166855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:46.167055 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:46.167210 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:16:46.167857 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.169323 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.172897 (+ 3574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.174100 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.174216 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.174467 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:46.174732 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 13:16:46.174895 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11592,"thread_start_us":11410,"threads_started":1}
I20260619 13:16:46.175303 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.154127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42847 (local address 127.0.0.1:39122)
0619 13:16:46.166380 (+ 12253us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:46.166417 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:46.166588 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:46.167251 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:46.167258 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:46.167289 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:16:46.167672 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:46.167683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.170369 (+ 2686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.170374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:46.172721 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:46.172737 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.174301 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.174307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:46.174507 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.174551 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 13:16:46.174661 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 13:16:46.174867 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11997,"mutex_wait_us":31,"thread_start_us":11913,"threads_started":1}
I20260619 13:16:46.176764 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.154801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.0.97.3:44113)
0619 13:16:46.167147 (+ 12346us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:46.167157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:46.167910 (+ 753us) server_negotiation.cc:407] Connection header received
0619 13:16:46.168058 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:46.168065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:46.168310 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:46.168490 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:16:46.169381 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.170881 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.173201 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.174573 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.174693 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.174921 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:46.176285 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0619 13:16:46.176448 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12140,"thread_start_us":12359,"threads_started":1}
I20260619 13:16:46.177115 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.154415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44113 (local address 127.0.0.1:57750)
0619 13:16:46.167784 (+ 13369us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:46.167801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:46.167926 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:46.168752 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:46.168757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:46.168781 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:16:46.169172 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:46.169182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.171025 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.171030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:46.173010 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:46.173023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.175533 (+ 2510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.175538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:46.175652 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.175682 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 13:16:46.175767 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 13:16:46.175897 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13250,"mutex_wait_us":43,"thread_start_us":13096,"threads_started":1}
I20260619 13:16:46.177980 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.168390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45285 (local address 127.0.0.1:46718)
0619 13:16:46.169794 (+ 1404us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:46.169811 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:46.169933 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:46.171100 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:46.171104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:46.171129 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:16:46.171604 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:46.171616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.173414 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.173419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:46.175281 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:46.175297 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.177235 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.177251 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:46.177405 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.177434 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 13:16:46.177524 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 13:16:46.177639 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1274,"mutex_wait_us":3,"thread_start_us":1144,"threads_started":1}
I20260619 13:16:46.178079 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.168654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46718 (local address 127.0.97.2:45285)
0619 13:16:46.170385 (+ 1731us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:46.170394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:46.170418 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:16:46.170497 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:46.170504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:46.170702 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:46.170878 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:16:46.171796 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.173283 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.175466 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.176816 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.176930 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.177099 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:46.177584 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 13:16:46.177741 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1568,"thread_start_us":1448,"threads_started":1}
I20260619 13:16:46.179443 14363 tablet_service.cc:1467] Processing CreateTablet for tablet 431679fce399495a8d5fe30a165b0a2e (DEFAULT_TABLE table=test-table [id=52f1efb2c1014f6f99dfac092ab8248d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:16:46.180125 14630 tablet_service.cc:1467] Processing CreateTablet for tablet 431679fce399495a8d5fe30a165b0a2e (DEFAULT_TABLE table=test-table [id=52f1efb2c1014f6f99dfac092ab8248d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:16:46.180856 14363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431679fce399495a8d5fe30a165b0a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:16:46.181399 14630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431679fce399495a8d5fe30a165b0a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:16:46.183194 14497 tablet_service.cc:1467] Processing CreateTablet for tablet 431679fce399495a8d5fe30a165b0a2e (DEFAULT_TABLE table=test-table [id=52f1efb2c1014f6f99dfac092ab8248d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:16:46.184420 14497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431679fce399495a8d5fe30a165b0a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:16:46.192711 14713 tablet_bootstrap.cc:492] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24: Bootstrap starting.
I20260619 13:16:46.195727 14713 tablet_bootstrap.cc:654] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24: Neither blocks nor log segments found. Creating new log.
I20260619 13:16:46.196687 14713 log.cc:824] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:46.197523 14714 tablet_bootstrap.cc:492] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415: Bootstrap starting.
I20260619 13:16:46.200253 14715 tablet_bootstrap.cc:492] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c: Bootstrap starting.
I20260619 13:16:46.201517 14714 tablet_bootstrap.cc:654] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415: Neither blocks nor log segments found. Creating new log.
I20260619 13:16:46.202885 14714 log.cc:824] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:46.202976 14715 tablet_bootstrap.cc:654] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c: Neither blocks nor log segments found. Creating new log.
I20260619 13:16:46.204007 14715 log.cc:824] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:46.205876 14714 tablet_bootstrap.cc:492] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415: No bootstrap required, opened a new log
I20260619 13:16:46.206033 14713 tablet_bootstrap.cc:492] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24: No bootstrap required, opened a new log
I20260619 13:16:46.206138 14714 ts_tablet_manager.cc:1397] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 13:16:46.206305 14713 ts_tablet_manager.cc:1397] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260619 13:16:46.206696 14715 tablet_bootstrap.cc:492] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c: No bootstrap required, opened a new log
I20260619 13:16:46.206923 14715 ts_tablet_manager.cc:1397] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 13:16:46.213397 14714 raft_consensus.cc:348] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:46.213738 14714 raft_consensus.cc:374] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:16:46.213824 14714 raft_consensus.cc:729] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6f26ac0e3bd4f7b94f37b5b0317f415, State: Initialized, Role: FOLLOWER
I20260619 13:16:46.214424 14714 consensus_queue.cc:260] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:46.216035 14714 ts_tablet_manager.cc:1428] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260619 13:16:46.216563 14713 raft_consensus.cc:348] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:46.216626 14715 raft_consensus.cc:348] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:46.216993 14715 raft_consensus.cc:374] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:16:46.216993 14713 raft_consensus.cc:374] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:16:46.217079 14713 raft_consensus.cc:729] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3853ee3fe7ba42d7be2b0686ff189e24, State: Initialized, Role: FOLLOWER
I20260619 13:16:46.217077 14715 raft_consensus.cc:729] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dfb376a3034440384ab83181d050d9c, State: Initialized, Role: FOLLOWER
I20260619 13:16:46.217623 14713 consensus_queue.cc:260] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:46.217623 14715 consensus_queue.cc:260] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:46.219285 14695 heartbeater.cc:503] Master 127.0.97.62:38653 was elected leader, sending a full tablet report...
I20260619 13:16:46.219409 14715 ts_tablet_manager.cc:1428] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260619 13:16:46.219815 14713 ts_tablet_manager.cc:1428] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 13:16:46.219961 14562 heartbeater.cc:503] Master 127.0.97.62:38653 was elected leader, sending a full tablet report...
I20260619 13:16:46.232057 14721 raft_consensus.cc:482] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:16:46.232331 14721 raft_consensus.cc:504] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:46.233970 14721 leader_election.cc:288] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6f26ac0e3bd4f7b94f37b5b0317f415 (127.0.97.1:42847), 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285)
I20260619 13:16:46.240615 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.234722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47907 (local address 127.0.97.2:45285)
0619 13:16:46.234878 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:46.234882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:46.234897 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:16:46.234948 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:46.234952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:46.235004 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:46.235077 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:16:46.235446 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.236377 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.238745 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.239844 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.239977 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.240077 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:46.240278 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 13:16:46.240418 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 13:16:46.240675 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.234202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45285 (local address 127.0.97.3:47907)
0619 13:16:46.234404 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:46.234423 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:46.234535 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:46.235118 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:46.235122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:46.235143 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:16:46.235320 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:46.235328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.236517 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.236521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:46.238591 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:46.238606 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.239996 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.240001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:46.240135 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.240160 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 13:16:46.240233 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 13:16:46.240355 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":82}
I20260619 13:16:46.241091 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.234349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47183 (local address 127.0.97.1:42847)
0619 13:16:46.234539 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:46.234545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:46.235103 (+ 558us) server_negotiation.cc:407] Connection header received
0619 13:16:46.235248 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:46.235252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:46.235309 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:46.235376 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:16:46.235837 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.236754 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.239017 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.240208 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.240332 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.240423 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:46.240789 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 13:16:46.240903 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:16:46.241714 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.234277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42847 (local address 127.0.97.3:47183)
0619 13:16:46.234997 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:46.235011 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:46.235128 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:46.235414 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:46.235418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:46.235443 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:16:46.235694 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:46.235701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.236889 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.236894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:46.238889 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:46.238900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.240413 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.240419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:46.240592 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.240623 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:16:46.241356 (+ 733us) client_negotiation.cc:240] Negotiation successful
0619 13:16:46.241461 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":45,"thread_start_us":414,"threads_started":1}
I20260619 13:16:46.241766 14517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431679fce399495a8d5fe30a165b0a2e" candidate_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dfb376a3034440384ab83181d050d9c" is_pre_election: true
I20260619 13:16:46.242362 14517 raft_consensus.cc:2454] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3853ee3fe7ba42d7be2b0686ff189e24 in term 0.
I20260619 13:16:46.242664 14383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431679fce399495a8d5fe30a165b0a2e" candidate_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" is_pre_election: true
I20260619 13:16:46.243001 14586 leader_election.cc:302] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dfb376a3034440384ab83181d050d9c, 3853ee3fe7ba42d7be2b0686ff189e24; no voters:
I20260619 13:16:46.243170 14383 raft_consensus.cc:2454] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3853ee3fe7ba42d7be2b0686ff189e24 in term 0.
I20260619 13:16:46.243371 14721 raft_consensus.cc:2781] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:16:46.243511 14721 raft_consensus.cc:482] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:16:46.243618 14721 raft_consensus.cc:3037] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:16:46.246034 14721 raft_consensus.cc:504] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:46.246660 14721 leader_election.cc:288] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [CANDIDATE]: Term 1 election: Requested vote from peers d6f26ac0e3bd4f7b94f37b5b0317f415 (127.0.97.1:42847), 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285)
I20260619 13:16:46.247066 14383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431679fce399495a8d5fe30a165b0a2e" candidate_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415"
I20260619 13:16:46.247213 14517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431679fce399495a8d5fe30a165b0a2e" candidate_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dfb376a3034440384ab83181d050d9c"
I20260619 13:16:46.247274 14383 raft_consensus.cc:3037] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:16:46.247409 14517 raft_consensus.cc:3037] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:16:46.249564 14383 raft_consensus.cc:2454] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3853ee3fe7ba42d7be2b0686ff189e24 in term 1.
I20260619 13:16:46.249791 14517 raft_consensus.cc:2454] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3853ee3fe7ba42d7be2b0686ff189e24 in term 1.
I20260619 13:16:46.249961 14584 leader_election.cc:302] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3853ee3fe7ba42d7be2b0686ff189e24, d6f26ac0e3bd4f7b94f37b5b0317f415; no voters:
I20260619 13:16:46.250272 14721 raft_consensus.cc:2781] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:16:46.250641 14721 raft_consensus.cc:686] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [term 1 LEADER]: Becoming Leader. State: Replica: 3853ee3fe7ba42d7be2b0686ff189e24, State: Running, Role: LEADER
I20260619 13:16:46.251083 14721 consensus_queue.cc:237] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:46.255151 14238 catalog_manager.cc:5495] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } health_report { overall_health: UNKNOWN } } }
I20260619 13:16:46.277484 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.272551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.0.97.3:44113)
0619 13:16:46.272739 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:46.272743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:46.272761 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:16:46.272873 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:46.272876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:46.272951 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:46.273068 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:16:46.273355 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.274107 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.275410 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.275901 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.276016 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.276447 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:16:46.276957 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:16:46.277080 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:46.277171 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 13:16:46.277236 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:16:46.285173 14383 raft_consensus.cc:1261] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Refusing update from remote peer 3853ee3fe7ba42d7be2b0686ff189e24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:16:46.285174 14517 raft_consensus.cc:1261] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c [term 1 FOLLOWER]: Refusing update from remote peer 3853ee3fe7ba42d7be2b0686ff189e24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:16:46.286176 14723 consensus_queue.cc:1035] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:16:46.286505 14721 consensus_queue.cc:1035] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:16:46.311018 14238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:16:46.311269 14238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:16:46.314673 14238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: Sending DeleteTablet for 3 replicas of tablet 431679fce399495a8d5fe30a165b0a2e
I20260619 13:16:46.315773 14363 tablet_service.cc:1514] Processing DeleteTablet for tablet 431679fce399495a8d5fe30a165b0a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:16:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39122
I20260619 13:16:46.316030 14497 tablet_service.cc:1514] Processing DeleteTablet for tablet 431679fce399495a8d5fe30a165b0a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:16:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46718
I20260619 13:16:46.316185 14630 tablet_service.cc:1514] Processing DeleteTablet for tablet 431679fce399495a8d5fe30a165b0a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:16:46 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57750
I20260619 13:16:46.317154 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 14300
I20260619 13:16:46.326875 14737 tablet_replica.cc:330] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24: stopping tablet replica
I20260619 13:16:46.327700 14737 raft_consensus.cc:2229] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:16:46.328492 14737 raft_consensus.cc:2258] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:16:46.329057 14735 tablet_replica.cc:330] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415: stopping tablet replica
I20260619 13:16:46.329425 14735 raft_consensus.cc:2229] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:16:46.329913 14735 raft_consensus.cc:2258] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:16:46.330786 14736 tablet_replica.cc:330] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c: stopping tablet replica
I20260619 13:16:46.331151 14736 raft_consensus.cc:2229] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:16:46.331557 14736 raft_consensus.cc:2258] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:16:46.333006 14737 ts_tablet_manager.cc:1905] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:16:46.338970 14737 ts_tablet_manager.cc:1918] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:16:46.339148 14737 log.cc:1196] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/431679fce399495a8d5fe30a165b0a2e
I20260619 13:16:46.339695 14737 ts_tablet_manager.cc:1939] T 431679fce399495a8d5fe30a165b0a2e P 3853ee3fe7ba42d7be2b0686ff189e24: Deleting consensus metadata
I20260619 13:16:46.340071 14736 ts_tablet_manager.cc:1905] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:16:46.340131 14735 ts_tablet_manager.cc:1905] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:16:46.341439 14226 catalog_manager.cc:4841] TS 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3:44113): tablet 431679fce399495a8d5fe30a165b0a2e (table test-table [id=52f1efb2c1014f6f99dfac092ab8248d]) successfully deleted
I20260619 13:16:46.346897 14736 ts_tablet_manager.cc:1918] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:16:46.347064 14736 log.cc:1196] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/431679fce399495a8d5fe30a165b0a2e
I20260619 13:16:46.347574 14736 ts_tablet_manager.cc:1939] T 431679fce399495a8d5fe30a165b0a2e P 1dfb376a3034440384ab83181d050d9c: Deleting consensus metadata
I20260619 13:16:46.347751 14735 ts_tablet_manager.cc:1918] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:16:46.347983 14735 log.cc:1196] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/431679fce399495a8d5fe30a165b0a2e
I20260619 13:16:46.348500 14735 ts_tablet_manager.cc:1939] T 431679fce399495a8d5fe30a165b0a2e P d6f26ac0e3bd4f7b94f37b5b0317f415: Deleting consensus metadata
Jun 19 13:16:46 dist-test-slave-mkt3 krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:16:46.349536 14226 catalog_manager.cc:4841] TS 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285): tablet 431679fce399495a8d5fe30a165b0a2e (table test-table [id=52f1efb2c1014f6f99dfac092ab8248d]) successfully deleted
I20260619 13:16:46.350313 14224 catalog_manager.cc:4841] TS d6f26ac0e3bd4f7b94f37b5b0317f415 (127.0.97.1:42847): tablet 431679fce399495a8d5fe30a165b0a2e (table test-table [id=52f1efb2c1014f6f99dfac092ab8248d]) successfully deleted
I20260619 13:16:46.355985 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.318318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39134 (local address 127.0.97.1:42847)
0619 13:16:46.318481 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:46.318487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:46.334543 (+ 16056us) server_negotiation.cc:407] Connection header received
0619 13:16:46.334755 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:46.334760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:46.334857 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:46.334958 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:46.336876 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.338014 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.342943 (+ 4929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.343594 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.343715 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.350720 (+ 7005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:46.350730 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:46.350744 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:46.350779 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:46.353353 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:46.354064 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:46.354071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:46.354073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:46.354198 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:46.354544 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:46.354547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:46.354549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:46.354888 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:46.355159 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:46.355468 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 13:16:46.355679 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:16:46.356614 14423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:46.506489 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 14300
I20260619 13:16:46.521656 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 14434
Jun 19 13:16:46 dist-test-slave-mkt3 krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:16:46.536927 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.522480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.0.97.2:45285)
0619 13:16:46.522664 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:46.522670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:46.522699 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:16:46.522818 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:46.522824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:46.522906 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:46.523016 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:46.524248 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.525344 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.526583 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.527208 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.527330 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.531657 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:46.531671 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:46.531686 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:46.531721 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:46.534474 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:46.535202 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:46.535208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:46.535211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:46.535320 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:46.535656 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:46.535659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:46.535661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:46.536007 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:46.536208 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:46.536510 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 13:16:46.536662 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:16:46.537519 14557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:46.648149 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 14434
I20260619 13:16:46.662456 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 14567
Jun 19 13:16:46 dist-test-slave-mkt3 krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:16:46.680058 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:46.663388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.0.97.3:44113)
0619 13:16:46.663549 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:46.663553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:46.663570 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:16:46.663727 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:46.663731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:46.663791 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:46.663877 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:46.665639 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.666462 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.668280 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:46.668713 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:46.668795 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:46.674275 (+ 5480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:46.674292 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:46.674307 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:46.674345 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:46.677167 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:46.677964 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:46.677972 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:46.677974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:46.678128 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:46.678537 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:46.678541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:46.678544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:46.678919 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:46.679148 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:46.679519 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 13:16:46.679686 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:16:46.680807 14690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:46.799772 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 14567
I20260619 13:16:46.815578 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 14207
I20260619 13:16:46.816635 14269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:46.922298 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 14207
I20260619 13:16:46.939005 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38653
--webserver_interface=127.0.97.62
--webserver_port=41293
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:16:47.247519 14743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:47.247789 14743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:47.247884 14743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:47.254885 14743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:16:47.254993 14743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:47.255049 14743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:47.255103 14743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:16:47.255146 14743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:16:47.264619 14743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=41293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:47.265775 14743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:47.266996 14743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:47.286909 14749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:47.286972 14752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:47.286936 14750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:47.287742 14743 server_base.cc:1034] running on GCE node
I20260619 13:16:47.288508 14743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:47.289975 14743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:47.291175 14743 hybrid_clock.cc:648] HybridClock initialized: now 1781875007291117 us; error 46 us; skew 500 ppm
I20260619 13:16:47.291538 14743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:47 dist-test-slave-mkt3 krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:47.294690 14743 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:16:47.296165 14743 webserver.cc:460] Webserver started at http://127.0.97.62:41293/ using document root <none> and password file <none>
I20260619 13:16:47.296852 14743 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:47.296975 14743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:47.302505 14743 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:16:47.305138 14759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:47.306535 14743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:16:47.306690 14743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "e3cebe8a96294ffcbd37162dcfd86876"
format_stamp: "Formatted at 2026-06-19 13:16:44 on dist-test-slave-mkt3"
server_key: "9f5eab2a4d547e1e9918724b40ccc111"
server_key_iv: "96e9e69e8486bb4e3200c9fdd1d17f16"
server_key_version: "encryptionkey@0"
I20260619 13:16:47.307684 14743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:47.323068 14743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:47.326840 14743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:47.327078 14743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:47.343956 14743 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38653
I20260619 13:16:47.343974 14810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38653 every 8 connection(s)
I20260619 13:16:47.345372 14743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:16:47.348193 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 14743
I20260619 13:16:47.349160 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42847
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38653
--webserver_port=36521
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:16:47.353822 14811 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:16:47.358022 14811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: Bootstrap starting.
I20260619 13:16:47.376758 14811 log.cc:824] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:47.388441 14811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:16:47.388942 14811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: Bootstrap complete.
I20260619 13:16:47.398083 14811 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } }
I20260619 13:16:47.398902 14811 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cebe8a96294ffcbd37162dcfd86876, State: Initialized, Role: FOLLOWER
I20260619 13:16:47.399520 14811 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } }
I20260619 13:16:47.399734 14811 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:16:47.399845 14811 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:16:47.400002 14811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:16:47.403687 14811 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } }
I20260619 13:16:47.404173 14811 leader_election.cc:302] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3cebe8a96294ffcbd37162dcfd86876; no voters:
I20260619 13:16:47.404760 14811 leader_election.cc:288] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 13:16:47.404883 14815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 13:16:47.406019 14815 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 2 LEADER]: Becoming Leader. State: Replica: e3cebe8a96294ffcbd37162dcfd86876, State: Running, Role: LEADER
I20260619 13:16:47.406637 14815 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } }
I20260619 13:16:47.407141 14811 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:16:47.409061 14817 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3cebe8a96294ffcbd37162dcfd86876. Latest consensus state: current_term: 2 leader_uuid: "e3cebe8a96294ffcbd37162dcfd86876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } } }
I20260619 13:16:47.409557 14817 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [sys.catalog]: This master's current role is: LEADER
I20260619 13:16:47.409579 14816 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e3cebe8a96294ffcbd37162dcfd86876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } } }
I20260619 13:16:47.409973 14816 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [sys.catalog]: This master's current role is: LEADER
I20260619 13:16:47.417649 14822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:16:47.426172 14822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:16:47.427831 14822 catalog_manager.cc:1259] Loaded cluster ID: 57362b2b1f3344fca7ea091f29005da8
I20260619 13:16:47.428017 14822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:16:47.432163 14822 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 13:16:47.451053 14822 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 13:16:47.458671 14822 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:16:47.459928 14822 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: Loaded TSK: 0
I20260619 13:16:47.461192 14822 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 13:16:47.697355 14813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:47.697639 14813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:47.697774 14813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:47.704813 14813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:47.704921 14813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:47.705061 14813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:16:47.714666 14813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36521
--tserver_master_addrs=127.0.97.62:38653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:47.715889 14813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:47.717164 14813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:47.730551 14838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:47.730492 14839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:47.731240 14841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:47.731866 14813 server_base.cc:1034] running on GCE node
I20260619 13:16:47.732600 14813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:47.733804 14813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:47.735086 14813 hybrid_clock.cc:648] HybridClock initialized: now 1781875007735023 us; error 48 us; skew 500 ppm
I20260619 13:16:47.735467 14813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:47 dist-test-slave-mkt3 krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:47.739400 14813 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:16:47.741117 14813 webserver.cc:460] Webserver started at http://127.0.97.1:36521/ using document root <none> and password file <none>
I20260619 13:16:47.742041 14813 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:47.742190 14813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:47.748998 14813 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:16:47.752277 14848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:47.753880 14813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:16:47.754052 14813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415"
format_stamp: "Formatted at 2026-06-19 13:16:45 on dist-test-slave-mkt3"
server_key: "426a65bee064dc6a8128e69f889afc62"
server_key_iv: "f7489ac7916a92fb0a908b887bd482be"
server_key_version: "encryptionkey@0"
I20260619 13:16:47.755380 14813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:47.791428 14813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:47.796469 14813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:47.796756 14813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:47.797945 14813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:16:47.799503 14813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:16:47.799616 14813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:47.799718 14813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:16:47.799777 14813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:47.835402 14813 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42847
I20260619 13:16:47.835457 14960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42847 every 8 connection(s)
I20260619 13:16:47.836796 14813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:16:47.842415 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 14813
I20260619 13:16:47.843114 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45285
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38653
--webserver_port=42365
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:16:47 dist-test-slave-mkt3 krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:47.860594 14964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:47.839622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:60419)
0619 13:16:47.840409 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:47.840461 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:47.842191 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:47.843763 (+ 1572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:47.843770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:47.844240 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:47.844769 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:47.844786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:47.846935 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:47.846938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:47.848118 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:47.848130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:47.849394 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:47.849400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:47.849629 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:47.850383 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:47.850412 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:47.852585 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:47.856490 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:47.856497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:47.856509 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:47.856942 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:47.857418 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:47.857421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:47.857423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:47.857592 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:47.858472 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:47.858477 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:47.858778 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 13:16:47.859026 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 13:16:47.859426 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":321,"threads_started":1}
I20260619 13:16:47.861012 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:47.839433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60419 (local address 127.0.97.62:38653)
0619 13:16:47.839970 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:47.839982 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:47.840587 (+ 605us) server_negotiation.cc:407] Connection header received
0619 13:16:47.842771 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:47.842792 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:47.843270 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:47.843653 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:47.845018 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:47.846794 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:47.848256 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:47.849182 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:47.849344 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:47.852856 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:47.852866 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:47.852882 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:47.852924 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:47.856337 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:47.857066 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:47.857075 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:47.857083 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:47.857321 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:47.857682 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:47.857686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:47.857689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:47.858352 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:47.858536 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:47.859262 (+ 726us) server_negotiation.cc:299] Negotiation successful
0619 13:16:47.859706 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":226,"threads_started":1}
I20260619 13:16:47.864065 14961 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38653
I20260619 13:16:47.864351 14961 heartbeater.cc:464] Registering TS with master...
I20260619 13:16:47.865005 14961 heartbeater.cc:511] Master 127.0.97.62:38653 requested a full tablet report, sending...
I20260619 13:16:47.867300 14776 ts_manager.cc:194] Registered new tserver with Master: d6f26ac0e3bd4f7b94f37b5b0317f415 (127.0.97.1:42847)
I20260619 13:16:47.869359 14776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60419
W20260619 13:16:48.158356 14965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:48.158635 14965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:48.158730 14965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:48.165920 14965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:48.166025 14965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:48.166143 14965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:16:48.175568 14965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42365
--tserver_master_addrs=127.0.97.62:38653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:48.176899 14965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:48.178510 14965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:48.191843 14971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:48.191874 14973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:48.191841 14970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:48.192505 14965 server_base.cc:1034] running on GCE node
I20260619 13:16:48.193142 14965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:48.194052 14965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:48.195238 14965 hybrid_clock.cc:648] HybridClock initialized: now 1781875008195182 us; error 47 us; skew 500 ppm
I20260619 13:16:48.195516 14965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:48 dist-test-slave-mkt3 krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:48.198483 14965 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:16:48.199848 14965 webserver.cc:460] Webserver started at http://127.0.97.2:42365/ using document root <none> and password file <none>
I20260619 13:16:48.200511 14965 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:48.200640 14965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:48.205909 14965 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:16:48.208397 14980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:48.209632 14965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:16:48.209777 14965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "1dfb376a3034440384ab83181d050d9c"
format_stamp: "Formatted at 2026-06-19 13:16:45 on dist-test-slave-mkt3"
server_key: "e3edc0d271864ff566c586e5ec969e0e"
server_key_iv: "0fb1ad2900922fbf52a0167e991bc31c"
server_key_version: "encryptionkey@0"
I20260619 13:16:48.210764 14965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:48.227718 14965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:48.231478 14965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:48.231717 14965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:48.232652 14965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:16:48.233918 14965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:16:48.234018 14965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:48.234120 14965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:16:48.234195 14965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:48.262849 14965 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45285
I20260619 13:16:48.262918 15092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45285 every 8 connection(s)
I20260619 13:16:48.264252 14965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:16:48.266582 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 14965
I20260619 13:16:48.267293 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44113
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38653
--webserver_port=34633
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:16:48 dist-test-slave-mkt3 krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:48.284834 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.266930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60659 (local address 127.0.97.62:38653)
0619 13:16:48.267107 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:48.267111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:48.268530 (+ 1419us) server_negotiation.cc:407] Connection header received
0619 13:16:48.270398 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:48.270405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:48.270489 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:48.270577 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:48.272109 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.272812 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.275030 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.275466 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.275572 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.279309 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:48.279314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:48.279317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:48.279354 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:48.281727 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:48.282580 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:48.282585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:48.282587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:48.282713 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:48.283140 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:48.283144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:48.283146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:48.283422 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:48.283570 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:48.284262 (+ 692us) server_negotiation.cc:299] Negotiation successful
0619 13:16:48.284498 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:16:48.285775 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.267591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:60659)
0619 13:16:48.268368 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:48.268405 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:48.270133 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:48.270842 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:48.270850 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:48.271362 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:48.271938 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:48.271955 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.272982 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.272987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:48.274877 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:48.274891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.275674 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.275680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:48.275959 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.276649 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:48.276676 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:48.279084 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:48.281917 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:48.281927 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:48.281945 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:48.282454 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:48.282825 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:48.282830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:48.282834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:48.283022 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:48.283546 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:48.283554 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:48.283894 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 13:16:48.284137 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 13:16:48.284676 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":320,"threads_started":1}
I20260619 13:16:48.287211 15093 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38653
I20260619 13:16:48.287577 15093 heartbeater.cc:464] Registering TS with master...
I20260619 13:16:48.288506 15093 heartbeater.cc:511] Master 127.0.97.62:38653 requested a full tablet report, sending...
I20260619 13:16:48.290191 14776 ts_manager.cc:194] Registered new tserver with Master: 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285)
I20260619 13:16:48.291013 14776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60659
W20260619 13:16:48.586346 15096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:48.586611 15096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:48.586742 15096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:48.593878 15096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:48.593990 15096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:48.594159 15096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:16:48.603943 15096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34633
--tserver_master_addrs=127.0.97.62:38653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:48.605146 15096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:48.606427 15096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:48.618798 15101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:48.618863 15102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:48.618836 15104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:48.625464 15096 server_base.cc:1034] running on GCE node
I20260619 13:16:48.626070 15096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:48.626955 15096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:48.628152 15096 hybrid_clock.cc:648] HybridClock initialized: now 1781875008628104 us; error 44 us; skew 500 ppm
I20260619 13:16:48.628453 15096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:48 dist-test-slave-mkt3 krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:48.635496 15096 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:16:48.636839 15096 webserver.cc:460] Webserver started at http://127.0.97.3:34633/ using document root <none> and password file <none>
I20260619 13:16:48.637493 15096 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:48.637612 15096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:48.643323 15096 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 13:16:48.646703 15111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:48.648473 15096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:16:48.648659 15096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "3853ee3fe7ba42d7be2b0686ff189e24"
format_stamp: "Formatted at 2026-06-19 13:16:46 on dist-test-slave-mkt3"
server_key: "370e28d5d3d969cebc320e10449b0c71"
server_key_iv: "468b3615154ccefadde62f4f5062a24f"
server_key_version: "encryptionkey@0"
I20260619 13:16:48.649787 15096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:48.666617 15096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:48.671382 15096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:48.671634 15096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:48.672585 15096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:16:48.673897 15096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:16:48.674001 15096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:48.674106 15096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:16:48.674180 15096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:48.702898 15096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44113
I20260619 13:16:48.702950 15223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44113 every 8 connection(s)
I20260619 13:16:48.704258 15096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:16:48.709071 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 15096
Jun 19 13:16:48 dist-test-slave-mkt3 krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:16:48.724557 15228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.710063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51296 (local address 127.0.97.62:38653)
0619 13:16:48.710764 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:48.710769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:48.710786 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:16:48.710833 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:48.710836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:48.710904 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:48.710986 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:48.712713 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.713797 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.715749 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.716294 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.716450 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.719239 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:48.719244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:48.719247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:48.719279 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:48.722095 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:48.722730 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:48.722734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:48.722739 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:48.722869 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:48.723210 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:48.723213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:48.723215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:48.723484 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:48.723719 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:48.724002 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 13:16:48.724209 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":497,"threads_started":1}
I20260619 13:16:48.725948 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.706599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47819 (local address 127.0.97.62:38653)
0619 13:16:48.706777 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:48.706781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:48.708033 (+ 1252us) server_negotiation.cc:407] Connection header received
0619 13:16:48.710264 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:48.710270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:48.710356 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:48.710429 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:48.712570 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.713780 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.716011 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.716631 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.716718 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.720426 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:48.720430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:48.720433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:48.720476 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:48.723011 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:48.723688 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:48.723692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:48.723694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:48.723798 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:48.724197 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:48.724202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:48.724206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:48.724517 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:48.724647 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:48.725392 (+ 745us) server_negotiation.cc:299] Negotiation successful
0619 13:16:48.725588 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":63}
I20260619 13:16:48.727398 15226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.707116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:47819)
0619 13:16:48.707864 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:48.707911 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:48.709985 (+ 2074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:48.710929 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:48.710937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:48.711603 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:48.712405 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:48.712429 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.713954 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.713960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:48.715866 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:48.715878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.716845 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.716853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:48.717120 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.717761 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:48.717783 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:48.720208 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:48.723149 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:48.723159 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:48.723175 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:48.723578 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:48.723908 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:48.723913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:48.723917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:48.724094 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:48.724644 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:48.724652 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:48.724966 (+ 314us) client_negotiation.cc:769] Sending connection context
0619 13:16:48.725287 (+ 321us) client_negotiation.cc:240] Negotiation successful
0619 13:16:48.725799 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":256,"threads_started":1}
I20260619 13:16:48.728667 15224 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38653
I20260619 13:16:48.728960 15224 heartbeater.cc:464] Registering TS with master...
I20260619 13:16:48.729617 15224 heartbeater.cc:511] Master 127.0.97.62:38653 requested a full tablet report, sending...
I20260619 13:16:48.730950 14776 ts_manager.cc:194] Registered new tserver with Master: 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3:44113)
I20260619 13:16:48.731715 14776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47819
I20260619 13:16:48.739063 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:16:48.745184 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.739856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.0.97.62:38653)
0619 13:16:48.740030 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:48.740037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:48.740443 (+ 406us) server_negotiation.cc:407] Connection header received
0619 13:16:48.740598 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:48.740602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:48.740733 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:48.740858 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:16:48.741214 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.742014 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.743452 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.743824 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.743926 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.744295 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:16:48.744754 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:16:48.744869 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:48.744909 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 13:16:48.744967 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:16:48.746013 14776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:16:48.752904 14776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:16:48.774107 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.766096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.0.97.2:45285)
0619 13:16:48.767024 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:48.767033 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:48.767064 (+ 31us) server_negotiation.cc:407] Connection header received
0619 13:16:48.767148 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:48.767153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:48.767305 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:48.767470 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:16:48.767986 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.769253 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.771499 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.772796 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.772920 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.773119 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:48.773577 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 13:16:48.773740 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":640,"threads_started":1}
I20260619 13:16:48.774173 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.765595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45285 (local address 127.0.0.1:46734)
0619 13:16:48.766436 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:48.766454 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:48.766552 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:48.767481 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:48.767486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:48.767510 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:16:48.767825 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:48.767834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.769448 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.769453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:48.771344 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:48.771357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.773222 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.773227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:48.773365 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.773402 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 13:16:48.773513 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 13:16:48.773701 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":49,"thread_start_us":773,"threads_started":1}
I20260619 13:16:48.775022 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.764030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.0.97.1:42847)
0619 13:16:48.764674 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:48.764683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:48.765394 (+ 711us) server_negotiation.cc:407] Connection header received
0619 13:16:48.766476 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:48.766484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:48.766735 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:48.766914 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:16:48.767591 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.769194 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.772247 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.773632 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.773737 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.773964 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:48.774513 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 13:16:48.774715 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":386,"threads_started":1}
I20260619 13:16:48.775170 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.764502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57778 (local address 127.0.97.3:44113)
0619 13:16:48.766026 (+ 1524us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:48.766034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:48.766066 (+ 32us) server_negotiation.cc:407] Connection header received
0619 13:16:48.766166 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:48.766173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:48.766459 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:48.767367 (+ 908us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:16:48.767642 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.769165 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.772247 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.773632 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.774428 (+ 796us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.774646 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:48.774714 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 13:16:48.774865 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1352,"thread_start_us":1697,"threads_started":1}
I20260619 13:16:48.774988 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.763806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42847 (local address 127.0.0.1:39144)
0619 13:16:48.765214 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:48.765286 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:48.765476 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:48.767027 (+ 1551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:48.767032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:48.767058 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:16:48.767402 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:48.767413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.769362 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.769368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:48.772073 (+ 2705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:48.772087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.774176 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.774183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:48.774336 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.774365 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 13:16:48.774444 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 13:16:48.774562 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1126,"mutex_wait_us":234,"spinlock_wait_cycles":896,"thread_start_us":1047,"threads_started":1}
I20260619 13:16:48.775053 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.763996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44113 (local address 127.0.0.1:57778)
0619 13:16:48.765214 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:48.765290 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:48.765475 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:48.766703 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:48.766712 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:48.766746 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:16:48.767126 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:48.767138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.769362 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.769368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:48.772073 (+ 2705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:48.772087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.773891 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.773898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:48.774084 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.774124 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 13:16:48.774188 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 13:16:48.774312 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":174,"thread_start_us":775,"threads_started":1}
I20260619 13:16:48.779294 14896 tablet_service.cc:1467] Processing CreateTablet for tablet fd273c852c3f439b915c68946c2017c7 (DEFAULT_TABLE table=test-table [id=a8d4a9bdbb034b5484a4c613d7647f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:16:48.779104 15028 tablet_service.cc:1467] Processing CreateTablet for tablet fd273c852c3f439b915c68946c2017c7 (DEFAULT_TABLE table=test-table [id=a8d4a9bdbb034b5484a4c613d7647f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:16:48.779104 15159 tablet_service.cc:1467] Processing CreateTablet for tablet fd273c852c3f439b915c68946c2017c7 (DEFAULT_TABLE table=test-table [id=a8d4a9bdbb034b5484a4c613d7647f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:16:48.780443 14896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd273c852c3f439b915c68946c2017c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:16:48.780529 15159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd273c852c3f439b915c68946c2017c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:16:48.780529 15028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd273c852c3f439b915c68946c2017c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:16:48.792750 15240 tablet_bootstrap.cc:492] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415: Bootstrap starting.
I20260619 13:16:48.796360 15240 tablet_bootstrap.cc:654] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415: Neither blocks nor log segments found. Creating new log.
I20260619 13:16:48.797379 15240 log.cc:824] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:48.798686 15241 tablet_bootstrap.cc:492] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c: Bootstrap starting.
I20260619 13:16:48.800431 15242 tablet_bootstrap.cc:492] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24: Bootstrap starting.
I20260619 13:16:48.800681 15240 tablet_bootstrap.cc:492] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415: No bootstrap required, opened a new log
I20260619 13:16:48.800978 15240 ts_tablet_manager.cc:1397] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260619 13:16:48.801983 15241 tablet_bootstrap.cc:654] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c: Neither blocks nor log segments found. Creating new log.
I20260619 13:16:48.803184 15241 log.cc:824] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:48.803217 15242 tablet_bootstrap.cc:654] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24: Neither blocks nor log segments found. Creating new log.
I20260619 13:16:48.804386 15242 log.cc:824] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:48.807066 15241 tablet_bootstrap.cc:492] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c: No bootstrap required, opened a new log
I20260619 13:16:48.807268 15242 tablet_bootstrap.cc:492] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24: No bootstrap required, opened a new log
I20260619 13:16:48.807309 15241 ts_tablet_manager.cc:1397] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 13:16:48.807508 15242 ts_tablet_manager.cc:1397] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260619 13:16:48.810873 15240 raft_consensus.cc:348] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:48.811297 15240 raft_consensus.cc:374] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:16:48.811405 15240 raft_consensus.cc:729] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6f26ac0e3bd4f7b94f37b5b0317f415, State: Initialized, Role: FOLLOWER
I20260619 13:16:48.811997 15240 consensus_queue.cc:260] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:48.813110 14961 heartbeater.cc:503] Master 127.0.97.62:38653 was elected leader, sending a full tablet report...
I20260619 13:16:48.813724 15240 ts_tablet_manager.cc:1428] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260619 13:16:48.815774 15241 raft_consensus.cc:348] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:48.816254 15241 raft_consensus.cc:374] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:16:48.816357 15241 raft_consensus.cc:729] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dfb376a3034440384ab83181d050d9c, State: Initialized, Role: FOLLOWER
I20260619 13:16:48.816764 15242 raft_consensus.cc:348] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:48.817068 15242 raft_consensus.cc:374] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:16:48.817167 15242 raft_consensus.cc:729] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3853ee3fe7ba42d7be2b0686ff189e24, State: Initialized, Role: FOLLOWER
I20260619 13:16:48.816938 15241 consensus_queue.cc:260] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:48.817657 15242 consensus_queue.cc:260] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:48.821592 15224 heartbeater.cc:503] Master 127.0.97.62:38653 was elected leader, sending a full tablet report...
I20260619 13:16:48.822412 15242 ts_tablet_manager.cc:1428] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260619 13:16:48.822494 15093 heartbeater.cc:503] Master 127.0.97.62:38653 was elected leader, sending a full tablet report...
I20260619 13:16:48.823110 15241 ts_tablet_manager.cc:1428] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
W20260619 13:16:48.838742 14962 tablet.cc:2369] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:16:48.906808 15248 raft_consensus.cc:482] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:16:48.907123 15248 raft_consensus.cc:504] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:48.908887 15248 leader_election.cc:288] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6f26ac0e3bd4f7b94f37b5b0317f415 (127.0.97.1:42847), 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285)
I20260619 13:16:48.914675 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.909397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42767 (local address 127.0.97.2:45285)
0619 13:16:48.909602 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:48.909609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:48.909633 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:16:48.909730 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:48.909736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:48.909811 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:48.909932 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:16:48.910351 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.911298 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.913042 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.913873 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.913982 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.914081 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:48.914312 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 13:16:48.914440 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:16:48.914688 15226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.909177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45285 (local address 127.0.97.3:42767)
0619 13:16:48.909375 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:48.909394 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:48.909516 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:48.909988 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:48.909993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:48.910025 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:16:48.910224 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:48.910235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.911434 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.911437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:48.912896 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:48.912909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.914007 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.914013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:48.914172 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.914197 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 13:16:48.914277 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 13:16:48.914401 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 13:16:48.915271 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.909490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32935 (local address 127.0.97.1:42847)
0619 13:16:48.909694 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:48.909700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:48.910136 (+ 436us) server_negotiation.cc:407] Connection header received
0619 13:16:48.910288 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:48.910294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:48.910367 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:48.910460 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:16:48.910941 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.911802 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.913452 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.914337 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.914452 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.914539 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:48.915010 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 13:16:48.915102 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:16:48.915401 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.909252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42847 (local address 127.0.97.3:32935)
0619 13:16:48.910031 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:48.910048 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:48.910165 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:48.910501 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:48.910505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:48.910527 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:16:48.910798 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:48.910806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.911947 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.911955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:48.913302 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:48.913313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.914578 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.914585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:48.914857 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.914889 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 13:16:48.914973 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 13:16:48.915086 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":481,"threads_started":1}
I20260619 13:16:48.915838 15048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd273c852c3f439b915c68946c2017c7" candidate_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dfb376a3034440384ab83181d050d9c" is_pre_election: true
I20260619 13:16:48.916337 15048 raft_consensus.cc:2454] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3853ee3fe7ba42d7be2b0686ff189e24 in term 0.
I20260619 13:16:48.916973 15115 leader_election.cc:302] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dfb376a3034440384ab83181d050d9c, 3853ee3fe7ba42d7be2b0686ff189e24; no voters:
I20260619 13:16:48.917309 15248 raft_consensus.cc:2781] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:16:48.917472 15248 raft_consensus.cc:482] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:16:48.917563 15248 raft_consensus.cc:3037] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:16:48.917534 14916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd273c852c3f439b915c68946c2017c7" candidate_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" is_pre_election: true
I20260619 13:16:48.918027 14916 raft_consensus.cc:2454] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3853ee3fe7ba42d7be2b0686ff189e24 in term 0.
I20260619 13:16:48.919883 15248 raft_consensus.cc:504] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:48.920430 15248 leader_election.cc:288] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [CANDIDATE]: Term 1 election: Requested vote from peers d6f26ac0e3bd4f7b94f37b5b0317f415 (127.0.97.1:42847), 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285)
I20260619 13:16:48.920781 14916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd273c852c3f439b915c68946c2017c7" candidate_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415"
I20260619 13:16:48.920879 15048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd273c852c3f439b915c68946c2017c7" candidate_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dfb376a3034440384ab83181d050d9c"
I20260619 13:16:48.920975 14916 raft_consensus.cc:3037] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:16:48.921058 15048 raft_consensus.cc:3037] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:16:48.924068 14916 raft_consensus.cc:2454] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3853ee3fe7ba42d7be2b0686ff189e24 in term 1.
I20260619 13:16:48.924067 15048 raft_consensus.cc:2454] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3853ee3fe7ba42d7be2b0686ff189e24 in term 1.
I20260619 13:16:48.924444 15113 leader_election.cc:302] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3853ee3fe7ba42d7be2b0686ff189e24, d6f26ac0e3bd4f7b94f37b5b0317f415; no voters:
I20260619 13:16:48.924741 15248 raft_consensus.cc:2781] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:16:48.925141 15248 raft_consensus.cc:686] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [term 1 LEADER]: Becoming Leader. State: Replica: 3853ee3fe7ba42d7be2b0686ff189e24, State: Running, Role: LEADER
I20260619 13:16:48.925645 15248 consensus_queue.cc:237] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:48.930294 14776 catalog_manager.cc:5495] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } health_report { overall_health: UNKNOWN } } }
W20260619 13:16:48.955884 15225 tablet.cc:2369] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:16:48.964859 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:48.959878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.0.97.3:44113)
0619 13:16:48.960044 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:48.960050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:48.960068 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:16:48.960163 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:48.960167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:48.960245 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:48.960363 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:16:48.960738 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.961493 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.962910 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:48.963294 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:48.963398 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:48.963853 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:16:48.964332 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:16:48.964455 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:48.964554 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 13:16:48.964631 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 13:16:48.972525 15048 raft_consensus.cc:1261] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c [term 1 FOLLOWER]: Refusing update from remote peer 3853ee3fe7ba42d7be2b0686ff189e24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:16:48.972525 14916 raft_consensus.cc:1261] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Refusing update from remote peer 3853ee3fe7ba42d7be2b0686ff189e24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:16:48.973503 15248 consensus_queue.cc:1035] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:16:48.973771 15250 consensus_queue.cc:1035] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:16:48.984460 15254 mvcc.cc:205] Tried to move back new op lower bound from 7298560036741386240 to 7298560036563488768. Current Snapshot: MvccSnapshot[applied={T|T < 7298560036741386240}]
I20260619 13:16:48.989130 15253 mvcc.cc:205] Tried to move back new op lower bound from 7298560036741386240 to 7298560036563488768. Current Snapshot: MvccSnapshot[applied={T|T < 7298560036741386240}]
I20260619 13:16:48.996487 14775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:16:48.996742 14775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:16:49.000255 14775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: Sending DeleteTablet for 3 replicas of tablet fd273c852c3f439b915c68946c2017c7
I20260619 13:16:49.001480 14896 tablet_service.cc:1514] Processing DeleteTablet for tablet fd273c852c3f439b915c68946c2017c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:16:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39144
I20260619 13:16:49.001528 15028 tablet_service.cc:1514] Processing DeleteTablet for tablet fd273c852c3f439b915c68946c2017c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:16:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46734
I20260619 13:16:49.001919 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 14813
I20260619 13:16:49.002595 15159 tablet_service.cc:1514] Processing DeleteTablet for tablet fd273c852c3f439b915c68946c2017c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:16:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57778
I20260619 13:16:49.017098 15263 tablet_replica.cc:330] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24: stopping tablet replica
I20260619 13:16:49.017750 15263 raft_consensus.cc:2229] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:16:49.018558 15263 raft_consensus.cc:2258] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:16:49.019389 15261 tablet_replica.cc:330] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c: stopping tablet replica
I20260619 13:16:49.019625 15262 tablet_replica.cc:330] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415: stopping tablet replica
I20260619 13:16:49.019835 15261 raft_consensus.cc:2229] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:16:49.019976 15262 raft_consensus.cc:2229] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:16:49.020355 15261 raft_consensus.cc:2258] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:16:49.020407 15262 raft_consensus.cc:2258] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:16:49.022439 15262 ts_tablet_manager.cc:1905] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:16:49.022439 15263 ts_tablet_manager.cc:1905] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:16:49.022893 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:49.003296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.0.97.1:42847)
0619 13:16:49.003484 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:49.003491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:49.003511 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:16:49.003573 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:49.003579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:49.003679 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:49.003798 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:49.005437 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:49.006696 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:49.008598 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:49.009433 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:49.009564 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:49.013444 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:49.013462 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:49.013480 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:49.013525 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:49.017633 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:49.019219 (+ 1586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:49.019229 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:49.019232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:49.019393 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:49.020155 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:49.020160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:49.020164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:49.020810 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:49.021076 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:49.022242 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0619 13:16:49.022456 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:16:49.023901 14956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:49.025929 15261 ts_tablet_manager.cc:1905] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:16:49.029446 15263 ts_tablet_manager.cc:1918] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:16:49.029624 15263 log.cc:1196] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/fd273c852c3f439b915c68946c2017c7
I20260619 13:16:49.030210 15263 ts_tablet_manager.cc:1939] T fd273c852c3f439b915c68946c2017c7 P 3853ee3fe7ba42d7be2b0686ff189e24: Deleting consensus metadata
I20260619 13:16:49.032004 14763 catalog_manager.cc:4841] TS 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3:44113): tablet fd273c852c3f439b915c68946c2017c7 (table test-table [id=a8d4a9bdbb034b5484a4c613d7647f17]) successfully deleted
I20260619 13:16:49.032406 15261 ts_tablet_manager.cc:1918] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:16:49.032550 15261 log.cc:1196] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/fd273c852c3f439b915c68946c2017c7
I20260619 13:16:49.032982 15261 ts_tablet_manager.cc:1939] T fd273c852c3f439b915c68946c2017c7 P 1dfb376a3034440384ab83181d050d9c: Deleting consensus metadata
I20260619 13:16:49.034365 14763 catalog_manager.cc:4841] TS 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285): tablet fd273c852c3f439b915c68946c2017c7 (table test-table [id=a8d4a9bdbb034b5484a4c613d7647f17]) successfully deleted
I20260619 13:16:49.122486 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 14813
I20260619 13:16:49.137254 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 14965
W20260619 13:16:49.137465 14761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42847 (error 108)
I20260619 13:16:49.139971 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:49.139498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42847 (local address 127.0.0.1:39150)
0619 13:16:49.139690 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:49.139803 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 13:16:49.140326 14761 catalog_manager.cc:4584] TS d6f26ac0e3bd4f7b94f37b5b0317f415 (127.0.97.1:42847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd273c852c3f439b915c68946c2017c7: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42847: connect: Connection refused (error 111)
I20260619 13:16:49.152576 15239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:49.138187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46736 (local address 127.0.97.2:45285)
0619 13:16:49.138393 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:49.138399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:49.138421 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:16:49.138570 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:49.138576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:49.138675 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:49.138770 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:49.140389 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:49.141506 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:49.142903 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:49.143535 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:49.143664 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:49.146171 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:49.146191 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:49.146210 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:49.146254 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:49.149720 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:49.150609 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:49.150618 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:49.150622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:49.150779 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:49.151108 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:49.151113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:49.151116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:49.151552 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:49.151836 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:49.151998 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 13:16:49.152212 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:16:49.153266 15088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:49.194018 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:49.193421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42847 (local address 127.0.0.1:39166)
0619 13:16:49.193675 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:49.193806 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 13:16:49.195223 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:49.194810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42847 (local address 127.0.0.1:39180)
0619 13:16:49.194973 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:49.195078 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 13:16:49.241616 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 14965
I20260619 13:16:49.255851 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 15096
I20260619 13:16:49.255992 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:49.255392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42847 (local address 127.0.0.1:39186)
0619 13:16:49.255693 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:49.255784 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260619 13:16:49.257376 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:49.257011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42847 (local address 127.0.0.1:39200)
0619 13:16:49.257168 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:49.257264 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 13:16:49.271008 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:49.256569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57794 (local address 127.0.97.3:44113)
0619 13:16:49.256765 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:49.256771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:49.256791 (+ 20us) server_negotiation.cc:407] Connection header received
0619 13:16:49.256951 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:49.256955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:49.257030 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:49.257119 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:49.259014 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:49.259817 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:49.261622 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:49.262135 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:49.262233 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:49.265378 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:49.265392 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:49.265409 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:49.265445 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:49.268258 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:49.269044 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:49.269050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:49.269052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:49.269196 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:49.269593 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:49.269599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:49.269601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:49.269997 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:49.270227 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:49.270546 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 13:16:49.270698 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:16:49.271688 15219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:49.338246 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:49.337746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42847 (local address 127.0.0.1:39212)
0619 13:16:49.337972 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:49.338085 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 13:16:49.339241 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:49.338872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42847 (local address 127.0.0.1:39226)
0619 13:16:49.339062 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:49.339136 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 13:16:49.388674 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 15096
I20260619 13:16:49.404556 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 14743
I20260619 13:16:49.405632 14806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:16:49.517903 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 14743
I20260619 13:16:49.517953 15235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:49.517418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42847 (local address 127.0.0.1:39230)
0619 13:16:49.517607 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:49.517711 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 13:16:49.535240 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38653
--webserver_interface=127.0.97.62
--webserver_port=41293
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:16:49.841976 15269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:49.842237 15269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:49.842329 15269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:49.849265 15269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:16:49.849355 15269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:49.849401 15269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:49.849454 15269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:16:49.849495 15269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:16:49.859071 15269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=41293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:49.860267 15269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:49.861539 15269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:49.872241 15274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:49.872176 15277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:49.872278 15275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:49.873111 15269 server_base.cc:1034] running on GCE node
I20260619 13:16:49.873795 15269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:49.875173 15269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:49.876374 15269 hybrid_clock.cc:648] HybridClock initialized: now 1781875009876310 us; error 48 us; skew 500 ppm
I20260619 13:16:49.876678 15269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:49 dist-test-slave-mkt3 krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:49.888716 15269 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:16:49.890372 15269 webserver.cc:460] Webserver started at http://127.0.97.62:41293/ using document root <none> and password file <none>
I20260619 13:16:49.891060 15269 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:49.891165 15269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:49.896553 15269 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 13:16:49.899281 15284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:49.900544 15269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:16:49.900686 15269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "e3cebe8a96294ffcbd37162dcfd86876"
format_stamp: "Formatted at 2026-06-19 13:16:44 on dist-test-slave-mkt3"
server_key: "9f5eab2a4d547e1e9918724b40ccc111"
server_key_iv: "96e9e69e8486bb4e3200c9fdd1d17f16"
server_key_version: "encryptionkey@0"
I20260619 13:16:49.901598 15269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:49.914976 15269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:49.919027 15269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:49.919242 15269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:49.936069 15269 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38653
I20260619 13:16:49.936100 15335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38653 every 8 connection(s)
I20260619 13:16:49.937470 15269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:16:49.943648 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 15269
I20260619 13:16:49.944237 15336 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:16:49.944598 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42847
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38653
--webserver_port=36521
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:16:49.948449 15336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: Bootstrap starting.
I20260619 13:16:49.968417 15336 log.cc:824] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:49.984140 15336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:16:49.984635 15336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876: Bootstrap complete.
I20260619 13:16:49.993399 15336 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } }
I20260619 13:16:49.994263 15336 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cebe8a96294ffcbd37162dcfd86876, State: Initialized, Role: FOLLOWER
I20260619 13:16:49.994838 15336 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } }
I20260619 13:16:49.995054 15336 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:16:49.995160 15336 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:16:49.995311 15336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 2 FOLLOWER]: Advancing to term 3
I20260619 13:16:49.998907 15336 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } }
I20260619 13:16:49.999368 15336 leader_election.cc:302] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3cebe8a96294ffcbd37162dcfd86876; no voters:
I20260619 13:16:49.999982 15336 leader_election.cc:288] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 13:16:50.000120 15340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 13:16:50.001094 15340 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [term 3 LEADER]: Becoming Leader. State: Replica: e3cebe8a96294ffcbd37162dcfd86876, State: Running, Role: LEADER
I20260619 13:16:50.001644 15340 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } }
I20260619 13:16:50.002131 15336 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:16:50.004211 15342 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3cebe8a96294ffcbd37162dcfd86876. Latest consensus state: current_term: 3 leader_uuid: "e3cebe8a96294ffcbd37162dcfd86876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } } }
I20260619 13:16:50.004658 15341 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e3cebe8a96294ffcbd37162dcfd86876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3cebe8a96294ffcbd37162dcfd86876" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38653 } } }
I20260619 13:16:50.004915 15342 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [sys.catalog]: This master's current role is: LEADER
I20260619 13:16:50.005225 15341 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3cebe8a96294ffcbd37162dcfd86876 [sys.catalog]: This master's current role is: LEADER
I20260619 13:16:50.007781 15347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:16:50.016602 15347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:16:50.018194 15347 catalog_manager.cc:1259] Loaded cluster ID: 57362b2b1f3344fca7ea091f29005da8
I20260619 13:16:50.018364 15347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 13:16:50.020457 15347 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa21b8ef9ad google::LogMessage::Fail() at ??:0
@ 0x7fa21b8f288a google::LogMessage::SendToLog() at ??:0
@ 0x7fa21b8ef4de google::LogMessage::Flush() at ??:0
@ 0x7fa21b8f0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa231322512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa23131f976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa2313a7610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa2313a7439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa21e53b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa21e53e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa21e53e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa21e519eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa2217f36db start_thread at ??:0
@ 0x7fa21a78961f clone at ??:0
W20260619 13:16:50.291430 15338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:50.291706 15338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:50.291838 15338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:50.299032 15338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:50.299157 15338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:50.299305 15338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:16:50.308806 15338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36521
--tserver_master_addrs=127.0.97.62:38653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:50.310096 15338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:50.311372 15338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:50.323534 15377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:50.323448 15380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:50.323513 15378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:50.324241 15338 server_base.cc:1034] running on GCE node
I20260619 13:16:50.324790 15338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:50.325649 15338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:50.326864 15338 hybrid_clock.cc:648] HybridClock initialized: now 1781875010326790 us; error 55 us; skew 500 ppm
I20260619 13:16:50.327178 15338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:50 dist-test-slave-mkt3 krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:50.330183 15338 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:16:50.331578 15338 webserver.cc:460] Webserver started at http://127.0.97.1:36521/ using document root <none> and password file <none>
I20260619 13:16:50.332219 15338 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:50.332319 15338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:50.337529 15338 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:16:50.340445 15387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:50.341708 15338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:16:50.341893 15338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415"
format_stamp: "Formatted at 2026-06-19 13:16:45 on dist-test-slave-mkt3"
server_key: "426a65bee064dc6a8128e69f889afc62"
server_key_iv: "f7489ac7916a92fb0a908b887bd482be"
server_key_version: "encryptionkey@0"
I20260619 13:16:50.342797 15338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:50.359227 15338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:50.363210 15338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:50.363447 15338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:50.364368 15338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:16:50.366012 15394 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 13:16:50.371201 15338 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 13:16:50.371294 15338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260619 13:16:50.371407 15338 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 13:16:50.374432 15338 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 13:16:50.374501 15338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.002s
I20260619 13:16:50.374804 15394 tablet_bootstrap.cc:492] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415: Bootstrap starting.
I20260619 13:16:50.382447 15394 log.cc:824] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415: Log is configured to *not* fsync() on all Append() calls
I20260619 13:16:50.394260 15394 tablet_bootstrap.cc:492] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:16:50.395031 15394 tablet_bootstrap.cc:492] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415: Bootstrap complete.
I20260619 13:16:50.395560 15394 ts_tablet_manager.cc:1397] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.001s
I20260619 13:16:50.405274 15394 raft_consensus.cc:348] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:50.406445 15394 raft_consensus.cc:729] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6f26ac0e3bd4f7b94f37b5b0317f415, State: Initialized, Role: FOLLOWER
I20260619 13:16:50.407117 15394 consensus_queue.cc:260] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:50.408529 15394 ts_tablet_manager.cc:1428] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 13:16:50.414780 15338 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42847
I20260619 13:16:50.414824 15502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42847 every 8 connection(s)
I20260619 13:16:50.415948 15338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:16:50.417593 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 15338
I20260619 13:16:50.418319 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45285
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38653
--webserver_port=42365
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:16:50.420939 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:50.418817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:37937)
0619 13:16:50.419672 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:50.419870 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":342,"threads_started":1}
W20260619 13:16:50.421618 15503 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38653 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38653: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
I20260619 13:16:50.422459 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:50.422112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:59107)
0619 13:16:50.422262 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:50.422338 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 13:16:50.423296 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:50.422949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:47487)
0619 13:16:50.423084 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:50.423161 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 13:16:50.423542 15503 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:16:50.726998 15506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:50.727253 15506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:50.727372 15506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:50.734298 15506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:50.734401 15506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:50.734543 15506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:16:50.743942 15506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45285
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42365
--tserver_master_addrs=127.0.97.62:38653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:50.745143 15506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:50.746423 15506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:50.758469 15511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:50.758440 15514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:50.758574 15512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:50.759472 15506 server_base.cc:1034] running on GCE node
I20260619 13:16:50.760016 15506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:50.760866 15506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:50.762053 15506 hybrid_clock.cc:648] HybridClock initialized: now 1781875010762012 us; error 30 us; skew 500 ppm
I20260619 13:16:50.762333 15506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:50 dist-test-slave-mkt3 krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:50.765355 15506 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:16:50.766654 15506 webserver.cc:460] Webserver started at http://127.0.97.2:42365/ using document root <none> and password file <none>
I20260619 13:16:50.767278 15506 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:50.767390 15506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:50.772336 15506 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 13:16:50.774899 15521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:50.776125 15506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:16:50.776263 15506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "1dfb376a3034440384ab83181d050d9c"
format_stamp: "Formatted at 2026-06-19 13:16:45 on dist-test-slave-mkt3"
server_key: "e3edc0d271864ff566c586e5ec969e0e"
server_key_iv: "0fb1ad2900922fbf52a0167e991bc31c"
server_key_version: "encryptionkey@0"
I20260619 13:16:50.777163 15506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:50.790019 15506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:50.794023 15506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:50.794253 15506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:50.795142 15506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:16:50.796355 15506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:16:50.796450 15506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:50.796548 15506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:16:50.796615 15506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:50.824923 15506 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45285
I20260619 13:16:50.824970 15633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45285 every 8 connection(s)
I20260619 13:16:50.826314 15506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:16:50.828879 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 15506
I20260619 13:16:50.829551 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44113
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38653
--webserver_port=34633
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:16:50.831251 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:50.829371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:51847)
0619 13:16:50.830225 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:50.830403 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":390,"threads_started":1}
W20260619 13:16:50.831984 15634 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38653 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38653: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
I20260619 13:16:50.832779 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:50.832468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:38937)
0619 13:16:50.832610 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:50.832678 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 13:16:50.833568 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:50.833287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:34945)
0619 13:16:50.833403 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:50.833470 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 13:16:50.833784 15634 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:16:51.135843 15637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:16:51.136104 15637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:16:51.136219 15637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:16:51.143316 15637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:16:51.143421 15637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:16:51.143564 15637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:16:51.153061 15637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34633
--tserver_master_addrs=127.0.97.62:38653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:16:51.154320 15637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:16:51.155534 15637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:16:51.167704 15643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:51.167845 15645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:16:51.167730 15642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:16:51.168346 15637 server_base.cc:1034] running on GCE node
I20260619 13:16:51.168860 15637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:16:51.169746 15637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:16:51.170934 15637 hybrid_clock.cc:648] HybridClock initialized: now 1781875011170882 us; error 46 us; skew 500 ppm
I20260619 13:16:51.171217 15637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:16:51 dist-test-slave-mkt3 krb5kdc[14194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:16:51.174101 15637 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:16:51.175446 15637 webserver.cc:460] Webserver started at http://127.0.97.3:34633/ using document root <none> and password file <none>
I20260619 13:16:51.176069 15637 fs_manager.cc:359] Metadata directory not provided
I20260619 13:16:51.176179 15637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:16:51.181246 15637 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:16:51.183838 15652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:51.185066 15637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:16:51.185215 15637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "3853ee3fe7ba42d7be2b0686ff189e24"
format_stamp: "Formatted at 2026-06-19 13:16:46 on dist-test-slave-mkt3"
server_key: "370e28d5d3d969cebc320e10449b0c71"
server_key_iv: "468b3615154ccefadde62f4f5062a24f"
server_key_version: "encryptionkey@0"
I20260619 13:16:51.186170 15637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:16:51.203025 15637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:16:51.206945 15637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:16:51.207180 15637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:16:51.208117 15637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:16:51.209371 15637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:16:51.209463 15637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:51.209565 15637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:16:51.209635 15637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:16:51.238936 15637 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44113
I20260619 13:16:51.239012 15764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44113 every 8 connection(s)
I20260619 13:16:51.240295 15637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:16:51.244827 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:51.243105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:56791)
0619 13:16:51.243789 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:51.243974 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":262,"threads_started":1}
W20260619 13:16:51.245512 15765 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38653 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:38653: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
I20260619 13:16:51.246294 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:51.246010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:41807)
0619 13:16:51.246132 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:51.246197 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 13:16:51.247074 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:51.246778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:48069)
0619 13:16:51.246900 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:51.246974 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 13:16:51.247273 15765 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 13:16:51.250519 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 15637
I20260619 13:16:51.424678 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:51.424015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:50529)
0619 13:16:51.424393 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:51.424478 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":178,"threads_started":1}
I20260619 13:16:51.580286 15770 raft_consensus.cc:482] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:16:51.580562 15770 raft_consensus.cc:504] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:51.582228 15770 leader_election.cc:288] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3:44113), 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285)
Jun 19 13:16:51 dist-test-slave-mkt3 krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
Jun 19 13:16:51 dist-test-slave-mkt3 krb5kdc[14194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:16:51.605326 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:51.582460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44113 (local address 127.0.97.1:52681)
0619 13:16:51.582642 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:51.582680 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:51.583945 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:51.587344 (+ 3399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:51.587354 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:51.588177 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:51.589117 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:51.589145 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:51.591399 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:51.591405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:51.593633 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:51.593648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:51.595016 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:51.595024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:51.595364 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:51.596171 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:51.596196 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:51.599184 (+ 2988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:51.602219 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:51.602222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:51.602225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:51.602663 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:51.603053 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:51.603057 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:51.603059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:51.603241 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:51.603929 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:51.603935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:51.604222 (+ 287us) client_negotiation.cc:769] Sending connection context
0619 13:16:51.604462 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 13:16:51.604932 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":276,"spinlock_wait_cycles":44032}
I20260619 13:16:51.605404 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:51.582805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52681 (local address 127.0.97.3:44113)
0619 13:16:51.583906 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:51.583923 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:51.583971 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:16:51.586342 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:51.586354 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:51.586875 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:51.587240 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:51.589312 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:51.591215 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:51.593807 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:51.594793 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:51.594974 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:51.599412 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:51.599419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:51.599430 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:51.599468 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:51.602095 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:51.602780 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:51.602785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:51.602790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:51.602939 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:51.603343 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:51.603346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:51.603348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:51.603787 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:51.603946 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:51.604682 (+ 736us) server_negotiation.cc:299] Negotiation successful
0619 13:16:51.605046 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":760,"threads_started":1}
I20260619 13:16:51.605600 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:51.583369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60451 (local address 127.0.97.2:45285)
0619 13:16:51.584373 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0619 13:16:51.584387 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:16:51.584773 (+ 386us) server_negotiation.cc:407] Connection header received
0619 13:16:51.586817 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:16:51.586828 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:16:51.587282 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:16:51.587624 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:16:51.589312 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:51.591215 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:51.593791 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:16:51.594793 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:16:51.594974 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:51.599138 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:16:51.599149 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:16:51.599167 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:16:51.599210 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:16:51.601951 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:51.602723 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:51.602733 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:51.602740 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:51.602918 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:16:51.603324 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:16:51.603329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:16:51.603332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:16:51.603788 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:16:51.603951 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 13:16:51.604836 (+ 885us) server_negotiation.cc:299] Negotiation successful
0619 13:16:51.605188 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":704,"threads_started":1}
I20260619 13:16:51.606449 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:51.582837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45285 (local address 127.0.97.1:60451)
0619 13:16:51.584661 (+ 1824us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:51.584679 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:16:51.584777 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:16:51.587594 (+ 2817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:16:51.587598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:16:51.588172 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:16:51.589117 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:51.589145 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:51.591404 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:51.591408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:51.593633 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:16:51.593648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:16:51.594982 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:16:51.594990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:16:51.595368 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:16:51.596160 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:16:51.596203 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:16:51.598852 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:16:51.602095 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:51.602104 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:51.602117 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:51.602599 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:51.603035 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:16:51.603040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:16:51.603044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:16:51.603226 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:16:51.603923 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:16:51.603931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:16:51.604217 (+ 286us) client_negotiation.cc:769] Sending connection context
0619 13:16:51.604462 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 13:16:51.606076 (+ 1614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1712,"mutex_wait_us":1518,"spinlock_wait_cycles":10752,"thread_start_us":1573,"threads_started":1}
W20260619 13:16:51.608506 15391 leader_election.cc:341] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3:44113): Not found: Tablet not found: fd273c852c3f439b915c68946c2017c7
W20260619 13:16:51.608785 15391 leader_election.cc:341] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285): Not found: Tablet not found: fd273c852c3f439b915c68946c2017c7
I20260619 13:16:51.608896 15391 leader_election.cc:302] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6f26ac0e3bd4f7b94f37b5b0317f415; no voters: 1dfb376a3034440384ab83181d050d9c, 3853ee3fe7ba42d7be2b0686ff189e24
I20260619 13:16:51.609261 15770 raft_consensus.cc:2726] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 13:16:51.835073 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:51.834287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:36605)
0619 13:16:51.834726 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:51.834855 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":216,"threads_started":1}
I20260619 13:16:52.248420 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:52.247724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:46163)
0619 13:16:52.248104 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:52.248219 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":181,"threads_started":1}
I20260619 13:16:52.426154 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:52.425435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:37939)
0619 13:16:52.425889 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:52.425974 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":229,"threads_started":1}
I20260619 13:16:52.836511 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:52.835876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:50417)
0619 13:16:52.836283 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:52.836365 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":212,"threads_started":1}
I20260619 13:16:53.249815 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:53.249149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:56067)
0619 13:16:53.249555 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:53.249629 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":203,"threads_started":1}
I20260619 13:16:53.427750 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:53.427034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:53897)
0619 13:16:53.427494 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:53.427579 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":244,"threads_started":1}
I20260619 13:16:53.620034 15780 raft_consensus.cc:482] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:16:53.620242 15780 raft_consensus.cc:504] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:53.620857 15780 leader_election.cc:288] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3:44113), 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285)
W20260619 13:16:53.621829 15391 leader_election.cc:341] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3:44113): Not found: Tablet not found: fd273c852c3f439b915c68946c2017c7
W20260619 13:16:53.622090 15391 leader_election.cc:341] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285): Not found: Tablet not found: fd273c852c3f439b915c68946c2017c7
I20260619 13:16:53.622185 15391 leader_election.cc:302] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6f26ac0e3bd4f7b94f37b5b0317f415; no voters: 1dfb376a3034440384ab83181d050d9c, 3853ee3fe7ba42d7be2b0686ff189e24
I20260619 13:16:53.622416 15780 raft_consensus.cc:2726] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 13:16:53.837939 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:53.837271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:50441)
0619 13:16:53.837671 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:53.837747 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":197,"threads_started":1}
I20260619 13:16:54.251322 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:54.250643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:53639)
0619 13:16:54.251075 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:54.251159 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":233,"threads_started":1}
I20260619 13:16:54.429337 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:54.428644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:47933)
0619 13:16:54.429074 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:54.429167 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":219,"threads_started":1}
I20260619 13:16:54.839429 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:54.838782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:46685)
0619 13:16:54.839171 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:54.839248 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":188,"threads_started":1}
I20260619 13:16:55.252856 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:55.252152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:54975)
0619 13:16:55.252588 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:55.252671 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":214,"threads_started":1}
I20260619 13:16:55.431102 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:55.430350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:36379)
0619 13:16:55.430842 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:55.430925 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":224,"threads_started":1}
I20260619 13:16:55.840907 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:55.840246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:35681)
0619 13:16:55.840664 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:55.840739 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":236,"threads_started":1}
I20260619 13:16:56.254367 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:56.253681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:36545)
0619 13:16:56.254126 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:56.254210 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":246,"threads_started":1}
I20260619 13:16:56.377774 15789 raft_consensus.cc:482] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:16:56.378005 15789 raft_consensus.cc:504] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:16:56.378639 15789 leader_election.cc:288] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3:44113), 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285)
W20260619 13:16:56.379583 15391 leader_election.cc:341] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3:44113): Not found: Tablet not found: fd273c852c3f439b915c68946c2017c7
W20260619 13:16:56.379802 15391 leader_election.cc:341] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285): Not found: Tablet not found: fd273c852c3f439b915c68946c2017c7
I20260619 13:16:56.379910 15391 leader_election.cc:302] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6f26ac0e3bd4f7b94f37b5b0317f415; no voters: 1dfb376a3034440384ab83181d050d9c, 3853ee3fe7ba42d7be2b0686ff189e24
I20260619 13:16:56.380159 15789 raft_consensus.cc:2726] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 13:16:56.432655 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:56.431916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:54997)
0619 13:16:56.432366 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:56.432449 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":215,"threads_started":1}
I20260619 13:16:56.842409 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:56.841704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:44999)
0619 13:16:56.842162 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:56.842235 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":260,"threads_started":1}
I20260619 13:16:57.255857 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:57.255161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:59265)
0619 13:16:57.255598 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:57.255676 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":238,"threads_started":1}
I20260619 13:16:57.434238 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:57.433423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:45901)
0619 13:16:57.433901 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:57.434004 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":236,"threads_started":1}
I20260619 13:16:57.843938 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:57.843192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:42051)
0619 13:16:57.843672 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:57.843756 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":216,"threads_started":1}
I20260619 13:16:58.257510 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:58.256690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:51385)
0619 13:16:58.257188 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:58.257279 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":229,"threads_started":1}
I20260619 13:16:58.435822 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:58.435126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:34553)
0619 13:16:58.435558 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:58.435642 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":221,"threads_started":1}
I20260619 13:16:58.845500 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:58.844824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:34925)
0619 13:16:58.845237 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:58.845319 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":196,"threads_started":1}
I20260619 13:16:59.259017 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:59.258369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:50711)
0619 13:16:59.258757 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:59.258843 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":196,"threads_started":1}
I20260619 13:16:59.437342 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:59.436639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:53703)
0619 13:16:59.437088 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:59.437166 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":232,"threads_started":1}
I20260619 13:16:59.847041 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:16:59.846334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:58919)
0619 13:16:59.846795 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 13:16:59.846884 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":214,"threads_started":1}
I20260619 13:17:00.239120 15801 raft_consensus.cc:482] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:17:00.239316 15801 raft_consensus.cc:504] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:17:00.240016 15801 leader_election.cc:288] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3:44113), 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285)
W20260619 13:17:00.240928 15391 leader_election.cc:341] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3:44113): Not found: Tablet not found: fd273c852c3f439b915c68946c2017c7
W20260619 13:17:00.241173 15391 leader_election.cc:341] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285): Not found: Tablet not found: fd273c852c3f439b915c68946c2017c7
I20260619 13:17:00.241258 15391 leader_election.cc:302] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6f26ac0e3bd4f7b94f37b5b0317f415; no voters: 1dfb376a3034440384ab83181d050d9c, 3853ee3fe7ba42d7be2b0686ff189e24
I20260619 13:17:00.241526 15801 raft_consensus.cc:2726] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 13:17:00.260439 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:00.259778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:35493)
0619 13:17:00.260203 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:00.260282 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":204,"threads_started":1}
I20260619 13:17:00.439251 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:00.438456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:47715)
0619 13:17:00.438967 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:00.439057 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":234,"threads_started":1}
I20260619 13:17:00.848529 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:00.847851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:55493)
0619 13:17:00.848257 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:00.848353 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":203,"threads_started":1}
I20260619 13:17:01.261941 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:01.261256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:34135)
0619 13:17:01.261651 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:01.261729 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":209,"threads_started":1}
I20260619 13:17:01.440771 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:01.440075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:56061)
0619 13:17:01.440499 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:01.440599 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":230,"threads_started":1}
I20260619 13:17:01.850030 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:01.849301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:57411)
0619 13:17:01.849749 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:01.849831 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":212,"threads_started":1}
I20260619 13:17:02.263669 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:02.262863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:37113)
0619 13:17:02.263345 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:02.263460 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":215,"threads_started":1}
I20260619 13:17:02.442277 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:02.441556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:58391)
0619 13:17:02.442034 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:02.442114 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":228,"threads_started":1}
I20260619 13:17:02.851538 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:02.850846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:41237)
0619 13:17:02.851290 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:02.851368 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":234,"threads_started":1}
I20260619 13:17:03.265198 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:03.264513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:46691)
0619 13:17:03.264942 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:03.265021 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":224,"threads_started":1}
I20260619 13:17:03.443763 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:03.443087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:36713)
0619 13:17:03.443496 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:03.443584 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":205,"threads_started":1}
I20260619 13:17:03.853055 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:03.852345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:50733)
0619 13:17:03.852811 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:03.852896 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":219,"threads_started":1}
I20260619 13:17:04.266868 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:04.266098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:56277)
0619 13:17:04.266550 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:04.266660 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":209,"threads_started":1}
I20260619 13:17:04.445292 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:04.444585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:37099)
0619 13:17:04.445017 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:04.445104 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":221,"threads_started":1}
I20260619 13:17:04.854655 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:04.853949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:33107)
0619 13:17:04.854378 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:04.854457 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":213,"threads_started":1}
I20260619 13:17:05.268425 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:05.267709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:43469)
0619 13:17:05.268147 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:05.268235 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":210,"threads_started":1}
I20260619 13:17:05.447039 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:05.446322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:49675)
0619 13:17:05.446793 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:05.446868 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":219,"threads_started":1}
I20260619 13:17:05.856392 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:05.855722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:45079)
0619 13:17:05.856142 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:05.856234 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":217,"threads_started":1}
I20260619 13:17:06.224287 15820 raft_consensus.cc:482] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:17:06.224505 15820 raft_consensus.cc:504] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6f26ac0e3bd4f7b94f37b5b0317f415" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42847 } } peers { permanent_uuid: "3853ee3fe7ba42d7be2b0686ff189e24" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44113 } } peers { permanent_uuid: "1dfb376a3034440384ab83181d050d9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45285 } }
I20260619 13:17:06.225178 15820 leader_election.cc:288] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3:44113), 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285)
W20260619 13:17:06.226215 15391 leader_election.cc:341] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3853ee3fe7ba42d7be2b0686ff189e24 (127.0.97.3:44113): Not found: Tablet not found: fd273c852c3f439b915c68946c2017c7
W20260619 13:17:06.226409 15391 leader_election.cc:341] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1dfb376a3034440384ab83181d050d9c (127.0.97.2:45285): Not found: Tablet not found: fd273c852c3f439b915c68946c2017c7
I20260619 13:17:06.226495 15391 leader_election.cc:302] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6f26ac0e3bd4f7b94f37b5b0317f415; no voters: 1dfb376a3034440384ab83181d050d9c, 3853ee3fe7ba42d7be2b0686ff189e24
I20260619 13:17:06.226724 15820 raft_consensus.cc:2726] T fd273c852c3f439b915c68946c2017c7 P d6f26ac0e3bd4f7b94f37b5b0317f415 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 13:17:06.262372 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 15338
I20260619 13:17:06.270110 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:06.269401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.3:51977)
0619 13:17:06.269828 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:06.269942 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":236,"threads_started":1}
I20260619 13:17:06.448709 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:06.447848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.1:33671)
0619 13:17:06.448359 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:06.448462 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":234,"threads_started":1}
I20260619 13:17:06.858114 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:06.857368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38653 (local address 127.0.97.2:54803)
0619 13:17:06.857815 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:06.857945 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:38653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":222,"threads_started":1}
I20260619 13:17:06.923584 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:06.263401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58532 (local address 127.0.97.1:42847)
0619 13:17:06.264012 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:06.264022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:06.906521 (+642499us) server_negotiation.cc:407] Connection header received
0619 13:17:06.906630 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:06.906638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:06.906898 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:06.907048 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:06.908420 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:06.909443 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:06.910672 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:06.911216 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:06.911306 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:06.918143 (+ 6837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:06.918157 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:06.918171 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:06.918214 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:06.921002 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:06.921679 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:06.921685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:06.921687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:06.921821 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:06.922194 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:06.922198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:06.922200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:06.922558 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:06.922786 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:06.923024 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 13:17:06.923235 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":243,"threads_started":1}
I20260619 13:17:06.924814 15498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:17:07.018179 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 15338
I20260619 13:17:07.032078 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 15506
I20260619 13:17:07.044659 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:07.032865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.0.97.2:45285)
0619 13:17:07.033387 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:07.033391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:07.033408 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:17:07.033462 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:07.033466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:07.033525 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:07.033614 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:07.035084 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:07.035883 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:07.037090 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:07.037482 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:07.037567 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:07.039933 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:07.039938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:07.039941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:07.039973 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:07.042400 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:07.043005 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:07.043008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:07.043010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:07.043140 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:07.043491 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:07.043493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:07.043495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:07.043729 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:07.043951 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:07.044161 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 13:17:07.044321 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":285,"threads_started":1}
I20260619 13:17:07.045351 15629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:17:07.130221 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 15506
I20260619 13:17:07.144776 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 15637
I20260619 13:17:07.157119 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:07.145582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.0.97.3:44113)
0619 13:17:07.146133 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:07.146138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:07.146155 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:17:07.146210 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:07.146214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:07.146275 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:07.146365 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:07.147600 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:07.148409 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:07.149501 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:07.149958 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:07.150066 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:07.152386 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:07.152391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:07.152394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:07.152425 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:07.154786 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:07.155442 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:07.155446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:07.155449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:07.155560 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:07.155860 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:07.155863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:07.155865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:07.156109 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:07.156309 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:07.156616 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 13:17:07.156770 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":314,"threads_started":1}
I20260619 13:17:07.157816 15760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:17:07.241066 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 15637
2026-06-19T13:17:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24282 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:17:07 dist-test-slave-mkt3 krb5kdc[15832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:17:07 dist-test-slave-mkt3 krb5kdc[15832](info): set up 2 sockets
Jun 19 13:17:07 dist-test-slave-mkt3 krb5kdc[15832](info): commencing operation
krb5kdc: starting...
W20260619 13:17:08.250156 388 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:17:08 dist-test-slave-mkt3 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:17:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:17:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:17:08.471573 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34051
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:17:08.801059 15845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:17:08.801353 15845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:17:08.801482 15845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:17:08.808590 15845 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 13:17:08.808689 15845 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 13:17:08.808730 15845 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 13:17:08.808800 15845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:17:08.808867 15845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:17:08.808962 15845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:17:08.809017 15845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:17:08.809092 15845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:17:08.819018 15845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:17:08.820328 15845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:17:08.821696 15845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:17:08.832867 15853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:08.832926 15851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:08.832947 15850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:17:08.833650 15845 server_base.cc:1034] running on GCE node
I20260619 13:17:08.834376 15845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:17:08.835660 15845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:17:08.836894 15845 hybrid_clock.cc:648] HybridClock initialized: now 1781875028836822 us; error 46 us; skew 500 ppm
I20260619 13:17:08.837194 15845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:17:08 dist-test-slave-mkt3 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:17:08.840101 15845 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:17:08.841430 15845 webserver.cc:460] Webserver started at http://127.0.97.62:43983/ using document root <none> and password file <none>
I20260619 13:17:08.842128 15845 fs_manager.cc:359] Metadata directory not provided
I20260619 13:17:08.842221 15845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:17:08.842509 15845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:17:08.845269 15845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "69f8e3f19c4f466cb939b9f878d6d375"
format_stamp: "Formatted at 2026-06-19 13:17:08 on dist-test-slave-mkt3"
server_key: "9ce14709b748722ec628c2c04da2862f"
server_key_iv: "dfd6dcd604480331bc94a94640dd1c75"
server_key_version: "encryptionkey@0"
I20260619 13:17:08.846046 15845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "69f8e3f19c4f466cb939b9f878d6d375"
format_stamp: "Formatted at 2026-06-19 13:17:08 on dist-test-slave-mkt3"
server_key: "9ce14709b748722ec628c2c04da2862f"
server_key_iv: "dfd6dcd604480331bc94a94640dd1c75"
server_key_version: "encryptionkey@0"
I20260619 13:17:08.850800 15845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 13:17:08.854051 15860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:08.855311 15845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:17:08.855504 15845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "69f8e3f19c4f466cb939b9f878d6d375"
format_stamp: "Formatted at 2026-06-19 13:17:08 on dist-test-slave-mkt3"
server_key: "9ce14709b748722ec628c2c04da2862f"
server_key_iv: "dfd6dcd604480331bc94a94640dd1c75"
server_key_version: "encryptionkey@0"
I20260619 13:17:08.855726 15845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:17:08.867730 15845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:17:08.871280 15845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:17:08.871503 15845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:17:08.888182 15845 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34051
I20260619 13:17:08.888206 15911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34051 every 8 connection(s)
I20260619 13:17:08.889494 15845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:17:08.891721 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 15845
I20260619 13:17:08.891909 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:17:08.892499 388 external_mini_cluster.cc:1428] Setting key b6cb6d239d625804ec02e8ea6788ac05
I20260619 13:17:08.893463 15912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:17:08.905184 15912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Bootstrap starting.
Jun 19 13:17:08 dist-test-slave-mkt3 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:17:08.908584 15912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Neither blocks nor log segments found. Creating new log.
I20260619 13:17:08.909658 15912 log.cc:824] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Log is configured to *not* fsync() on all Append() calls
I20260619 13:17:08.912364 15912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: No bootstrap required, opened a new log
I20260619 13:17:08.913492 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:08.894380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.0.97.62:34051)
0619 13:17:08.895118 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:08.895130 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:08.895173 (+ 43us) server_negotiation.cc:407] Connection header received
0619 13:17:08.896728 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:08.896749 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:08.897148 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:08.897531 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:08.899001 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:08.900294 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:08.901576 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:08.902288 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:08.902479 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:08.906591 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:08.906600 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:08.906613 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:08.906658 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:08.909569 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:08.910347 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:08.910357 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:08.910363 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:08.910528 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:08.910895 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:08.910898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:08.910900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:08.911404 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:08.911666 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:08.912087 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 13:17:08.912510 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":323,"threads_started":1}
I20260619 13:17:08.920037 15912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 13:17:08.920440 15912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:17:08.920528 15912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f8e3f19c4f466cb939b9f878d6d375, State: Initialized, Role: FOLLOWER
I20260619 13:17:08.921056 15912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 13:17:08.921274 15912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:17:08.921370 15912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:17:08.921504 15912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:17:08.923813 15912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 13:17:08.924247 15912 leader_election.cc:302] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69f8e3f19c4f466cb939b9f878d6d375; no voters:
I20260619 13:17:08.924715 15912 leader_election.cc:288] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:17:08.924829 15917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:17:08.925433 15917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 1 LEADER]: Becoming Leader. State: Replica: 69f8e3f19c4f466cb939b9f878d6d375, State: Running, Role: LEADER
I20260619 13:17:08.925981 15917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 13:17:08.926617 15912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:17:08.928691 15918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69f8e3f19c4f466cb939b9f878d6d375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } } }
I20260619 13:17:08.928968 15918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [sys.catalog]: This master's current role is: LEADER
I20260619 13:17:08.930434 15924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:17:08.930440 15919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69f8e3f19c4f466cb939b9f878d6d375. Latest consensus state: current_term: 1 leader_uuid: "69f8e3f19c4f466cb939b9f878d6d375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } } }
I20260619 13:17:08.930728 15919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [sys.catalog]: This master's current role is: LEADER
I20260619 13:17:08.936090 15924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:17:08.944710 15924 catalog_manager.cc:1347] Generated new cluster ID: fa2394cc402c44868c2f6d2043be19fb
I20260619 13:17:08.944818 15924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:17:08.957942 15924 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:17:08.959357 15924 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:17:08.971128 15924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Generated new TSK 0
I20260619 13:17:08.972087 15924 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:17:09.035670 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34051
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:17:09.361795 15938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:17:09.362069 15938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:17:09.362164 15938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:17:09.369019 15938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:17:09.369107 15938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:17:09.369220 15938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:17:09.378357 15938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:17:09.379412 15938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:17:09.380570 15938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:17:09.392238 15946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:09.392277 15944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:09.392505 15943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:17:09.393183 15938 server_base.cc:1034] running on GCE node
I20260619 13:17:09.393709 15938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:17:09.394562 15938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:17:09.395753 15938 hybrid_clock.cc:648] HybridClock initialized: now 1781875029395708 us; error 35 us; skew 500 ppm
I20260619 13:17:09.396030 15938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:17:09 dist-test-slave-mkt3 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:17:09.398887 15938 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:17:09.400329 15938 webserver.cc:460] Webserver started at http://127.0.97.1:42301/ using document root <none> and password file <none>
I20260619 13:17:09.400961 15938 fs_manager.cc:359] Metadata directory not provided
I20260619 13:17:09.401075 15938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:17:09.401378 15938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:17:09.404134 15938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "4ca826eca75744778b8e2d48a188dc06"
format_stamp: "Formatted at 2026-06-19 13:17:09 on dist-test-slave-mkt3"
server_key: "0b04202f7bdd8db35ebbbdcd19b4a21a"
server_key_iv: "f9ec8d1ddb5dcf1d218d266982690e55"
server_key_version: "encryptionkey@0"
I20260619 13:17:09.404831 15938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4ca826eca75744778b8e2d48a188dc06"
format_stamp: "Formatted at 2026-06-19 13:17:09 on dist-test-slave-mkt3"
server_key: "0b04202f7bdd8db35ebbbdcd19b4a21a"
server_key_iv: "f9ec8d1ddb5dcf1d218d266982690e55"
server_key_version: "encryptionkey@0"
I20260619 13:17:09.409642 15938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 13:17:09.412997 15953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:09.414351 15938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:17:09.414484 15938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "4ca826eca75744778b8e2d48a188dc06"
format_stamp: "Formatted at 2026-06-19 13:17:09 on dist-test-slave-mkt3"
server_key: "0b04202f7bdd8db35ebbbdcd19b4a21a"
server_key_iv: "f9ec8d1ddb5dcf1d218d266982690e55"
server_key_version: "encryptionkey@0"
I20260619 13:17:09.414666 15938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:17:09.440970 15938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:17:09.444823 15938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:17:09.445043 15938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:17:09.445941 15938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:17:09.447474 15938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:17:09.447542 15938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:09.447649 15938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:17:09.447700 15938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:09.477658 15938 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44927
I20260619 13:17:09.477713 16065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44927 every 8 connection(s)
I20260619 13:17:09.478896 15938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:17:09.479512 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 15938
I20260619 13:17:09.479727 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:17:09.480453 388 external_mini_cluster.cc:1428] Setting key 212e0a0551f7a799749197e7339e8830
Jun 19 13:17:09 dist-test-slave-mkt3 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:17:09.498178 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:09.481474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57915 (local address 127.0.97.62:34051)
0619 13:17:09.481956 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:09.481961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:09.482867 (+ 906us) server_negotiation.cc:407] Connection header received
0619 13:17:09.484676 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:09.484681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:09.484762 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:09.484855 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:09.486372 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:09.487057 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:09.488575 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:09.488977 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:09.489083 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:09.492434 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:09.492439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:09.492442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:09.492473 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:09.495258 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:09.495935 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:09.495940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:09.495944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:09.496082 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:09.496445 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:09.496450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:09.496453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:09.496774 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:09.496913 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:09.497538 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 13:17:09.497770 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":216,"threads_started":1}
I20260619 13:17:09.498942 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:09.482013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:57915)
0619 13:17:09.482728 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:09.482771 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:09.484408 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:09.485089 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:09.485098 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:09.485596 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:17:09.486211 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:09.486229 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:09.487208 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:09.487212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:09.488435 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:09.488452 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:09.489185 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:09.489193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:09.489405 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:09.490045 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:17:09.490068 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:17:09.492233 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:17:09.495396 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:17:09.495402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:17:09.495414 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:17:09.495822 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:17:09.496171 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:17:09.496174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:17:09.496177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:17:09.496342 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:17:09.496874 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:17:09.496880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:17:09.497168 (+ 288us) client_negotiation.cc:769] Sending connection context
0619 13:17:09.497380 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 13:17:09.497753 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":288,"threads_started":1}
I20260619 13:17:09.500391 16066 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34051
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:17:09.500689 16066 heartbeater.cc:464] Registering TS with master...
I20260619 13:17:09.501369 16066 heartbeater.cc:511] Master 127.0.97.62:34051 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 13:17:09.503399 15877 ts_manager.cc:194] Registered new tserver with Master: 4ca826eca75744778b8e2d48a188dc06 (127.0.97.1:44927)
I20260619 13:17:09.504814 15877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:17:09.525388 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34051
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:17:09.827623 16072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:17:09.827876 16072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:17:09.827993 16072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:17:09.834966 16072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:17:09.835059 16072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:17:09.835178 16072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:17:09.844297 16072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:17:09.845374 16072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:17:09.846544 16072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:17:09.858145 16077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:09.858225 16078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:09.858192 16080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:17:09.859236 16072 server_base.cc:1034] running on GCE node
I20260619 13:17:09.859782 16072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:17:09.860956 16072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:17:09.862146 16072 hybrid_clock.cc:648] HybridClock initialized: now 1781875029862083 us; error 54 us; skew 500 ppm
I20260619 13:17:09.862447 16072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:17:09 dist-test-slave-mkt3 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:17:09.865252 16072 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:17:09.866572 16072 webserver.cc:460] Webserver started at http://127.0.97.2:36961/ using document root <none> and password file <none>
I20260619 13:17:09.867192 16072 fs_manager.cc:359] Metadata directory not provided
I20260619 13:17:09.867296 16072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:17:09.867574 16072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:17:09.870317 16072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "187b99c27439439b926caaa279a244ea"
format_stamp: "Formatted at 2026-06-19 13:17:09 on dist-test-slave-mkt3"
server_key: "dd6d0c1dbbeacc123f804406997c1ddd"
server_key_iv: "0df1b8017ca7ed6d49531e6aadcc9e88"
server_key_version: "encryptionkey@0"
I20260619 13:17:09.870994 16072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "187b99c27439439b926caaa279a244ea"
format_stamp: "Formatted at 2026-06-19 13:17:09 on dist-test-slave-mkt3"
server_key: "dd6d0c1dbbeacc123f804406997c1ddd"
server_key_iv: "0df1b8017ca7ed6d49531e6aadcc9e88"
server_key_version: "encryptionkey@0"
I20260619 13:17:09.875645 16072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:17:09.878810 16087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:09.880018 16072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 13:17:09.880151 16072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "187b99c27439439b926caaa279a244ea"
format_stamp: "Formatted at 2026-06-19 13:17:09 on dist-test-slave-mkt3"
server_key: "dd6d0c1dbbeacc123f804406997c1ddd"
server_key_iv: "0df1b8017ca7ed6d49531e6aadcc9e88"
server_key_version: "encryptionkey@0"
I20260619 13:17:09.880319 16072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:17:09.890547 16072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:17:09.894325 16072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:17:09.894565 16072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:17:09.895534 16072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:17:09.896726 16072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:17:09.896793 16072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:09.896904 16072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:17:09.896950 16072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:09.926203 16072 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37467
I20260619 13:17:09.926251 16199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37467 every 8 connection(s)
I20260619 13:17:09.927479 16072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:17:09.933616 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 16072
I20260619 13:17:09.933815 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:17:09.934372 388 external_mini_cluster.cc:1428] Setting key f747263791c0e63815aa6e2cb35637f7
Jun 19 13:17:09 dist-test-slave-mkt3 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:17:09.948662 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:09.929921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38495 (local address 127.0.97.62:34051)
0619 13:17:09.930118 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:09.930125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:09.931298 (+ 1173us) server_negotiation.cc:407] Connection header received
0619 13:17:09.933018 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:09.933022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:09.933089 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:09.933171 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:09.935221 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:09.936212 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:09.938352 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:09.938750 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:09.938845 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:09.942658 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:09.942663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:09.942666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:09.942696 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:09.945441 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:09.946157 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:09.946160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:09.946162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:09.946290 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:09.946685 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:09.946689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:09.946691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:09.946954 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:09.947082 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:09.947927 (+ 845us) server_negotiation.cc:299] Negotiation successful
0619 13:17:09.948210 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:17:09.949786 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:09.930456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:38495)
0619 13:17:09.931140 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:09.931178 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:09.932773 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:09.933444 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:09.933457 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:09.934257 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:17:09.935051 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:09.935073 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:09.936351 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:09.936356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:09.938214 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:09.938235 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:09.938951 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:09.938957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:09.939208 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:09.939870 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:17:09.939896 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:17:09.942438 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:17:09.945552 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:17:09.945559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:17:09.945572 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:17:09.946035 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:17:09.946392 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:17:09.946397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:17:09.946401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:17:09.946572 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:17:09.947073 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:17:09.947080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:17:09.947456 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 13:17:09.947753 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 13:17:09.948277 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":258,"threads_started":1}
I20260619 13:17:09.951864 16200 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34051
I20260619 13:17:09.952216 16200 heartbeater.cc:464] Registering TS with master...
I20260619 13:17:09.953090 16200 heartbeater.cc:511] Master 127.0.97.62:34051 requested a full tablet report, sending...
I20260619 13:17:09.956820 15877 ts_manager.cc:194] Registered new tserver with Master: 187b99c27439439b926caaa279a244ea (127.0.97.2:37467)
I20260619 13:17:09.957924 15877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38495
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:17:09.986425 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34051
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:17:10.290593 16205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:17:10.290863 16205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:17:10.290994 16205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:17:10.297922 16205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:17:10.298017 16205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:17:10.298157 16205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:17:10.307319 16205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:17:10.308460 16205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:17:10.309636 16205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:17:10.321295 16210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:10.321256 16213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:10.321345 16211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:17:10.322386 16205 server_base.cc:1034] running on GCE node
I20260619 13:17:10.322926 16205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:17:10.323733 16205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:17:10.324920 16205 hybrid_clock.cc:648] HybridClock initialized: now 1781875030324872 us; error 42 us; skew 500 ppm
I20260619 13:17:10.325197 16205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:17:10 dist-test-slave-mkt3 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:17:10.327986 16205 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:17:10.329523 16205 webserver.cc:460] Webserver started at http://127.0.97.3:45479/ using document root <none> and password file <none>
I20260619 13:17:10.330160 16205 fs_manager.cc:359] Metadata directory not provided
I20260619 13:17:10.330257 16205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:17:10.330506 16205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:17:10.333192 16205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "27f18f77dbd74c30a72c4d048de58669"
format_stamp: "Formatted at 2026-06-19 13:17:10 on dist-test-slave-mkt3"
server_key: "771c5a9accb251e73fcfc85a51ca907e"
server_key_iv: "5c44bf437847cb2e438d3f8dfb6dc2f0"
server_key_version: "encryptionkey@0"
I20260619 13:17:10.333988 16205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "27f18f77dbd74c30a72c4d048de58669"
format_stamp: "Formatted at 2026-06-19 13:17:10 on dist-test-slave-mkt3"
server_key: "771c5a9accb251e73fcfc85a51ca907e"
server_key_iv: "5c44bf437847cb2e438d3f8dfb6dc2f0"
server_key_version: "encryptionkey@0"
I20260619 13:17:10.338867 16205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:17:10.342111 16220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:10.343429 16205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:17:10.343556 16205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "27f18f77dbd74c30a72c4d048de58669"
format_stamp: "Formatted at 2026-06-19 13:17:10 on dist-test-slave-mkt3"
server_key: "771c5a9accb251e73fcfc85a51ca907e"
server_key_iv: "5c44bf437847cb2e438d3f8dfb6dc2f0"
server_key_version: "encryptionkey@0"
I20260619 13:17:10.343712 16205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:17:10.362331 16205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:17:10.366118 16205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:17:10.366353 16205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:17:10.367229 16205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:17:10.368394 16205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:17:10.368462 16205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:10.368577 16205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:17:10.368625 16205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:10.395669 16205 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38541
I20260619 13:17:10.395713 16332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38541 every 8 connection(s)
I20260619 13:17:10.396817 16205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:17:10.403908 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 16205
I20260619 13:17:10.404093 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:17:10.404493 388 external_mini_cluster.cc:1428] Setting key 5d3670b0e6987bcd15e5e2707be0ba54
Jun 19 13:17:10 dist-test-slave-mkt3 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:17:10.415135 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.399123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39079 (local address 127.0.97.62:34051)
0619 13:17:10.399308 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:10.399312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:10.400449 (+ 1137us) server_negotiation.cc:407] Connection header received
0619 13:17:10.402292 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:10.402297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:10.402377 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:10.402461 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:10.404040 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.404808 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.406646 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.407103 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.407191 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.410302 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:10.410307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:10.410310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:10.410344 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:10.412612 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:10.413250 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:10.413254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:10.413256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:10.413365 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:10.413704 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:10.413707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:10.413709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:10.413991 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:10.414093 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:10.414719 (+ 626us) server_negotiation.cc:299] Negotiation successful
0619 13:17:10.414897 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:17:10.416059 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.399606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:39079)
0619 13:17:10.400299 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:10.400336 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:10.402041 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:10.402690 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:10.402698 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:10.403184 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:17:10.403852 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:10.403880 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.404961 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.404965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:10.406515 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:10.406529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.407317 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.407325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:10.407535 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.408103 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:17:10.408128 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:17:10.410125 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:17:10.412752 (+ 2627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:17:10.412758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:17:10.412770 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:17:10.413161 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:17:10.413449 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:17:10.413452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:17:10.413454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:17:10.413625 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:17:10.414090 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:17:10.414096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:17:10.414403 (+ 307us) client_negotiation.cc:769] Sending connection context
0619 13:17:10.414621 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 13:17:10.415009 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":263,"threads_started":1}
I20260619 13:17:10.417366 16333 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34051
I20260619 13:17:10.417626 16333 heartbeater.cc:464] Registering TS with master...
I20260619 13:17:10.418449 16333 heartbeater.cc:511] Master 127.0.97.62:34051 requested a full tablet report, sending...
I20260619 13:17:10.419807 15877 ts_manager.cc:194] Registered new tserver with Master: 27f18f77dbd74c30a72c4d048de58669 (127.0.97.3:38541)
I20260619 13:17:10.420564 15877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39079
I20260619 13:17:10.431527 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:17:10.447257 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.435871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45410 (local address 127.0.97.62:34051)
0619 13:17:10.436088 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:10.436095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:10.436396 (+ 301us) server_negotiation.cc:407] Connection header received
0619 13:17:10.436521 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:10.436525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:10.436586 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:10.436660 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:10.438042 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.438731 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.440025 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.440395 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.440475 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.442693 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:10.442697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:10.442701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:10.442730 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:10.445030 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:10.445589 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:10.445593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:10.445595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:10.445698 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:10.446019 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:10.446021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:10.446024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:10.446333 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:10.446500 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:10.446785 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 13:17:10.446969 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 13:17:10.450901 15877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:17:10.457620 15877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:17:10.494625 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.480573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38541 (local address 127.0.0.1:37226)
0619 13:17:10.487048 (+ 6475us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:10.487065 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:10.487173 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:10.487725 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:10.487729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:10.487757 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:17:10.488407 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:10.488424 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.490177 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.490182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:10.492393 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:10.492408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.493623 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.493629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:10.493794 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.493865 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 13:17:10.493984 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 13:17:10.494163 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6339,"mutex_wait_us":83,"thread_start_us":6381,"threads_started":1}
I20260619 13:17:10.495260 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.469071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37467 (local address 127.0.0.1:42828)
0619 13:17:10.485453 (+ 16382us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:10.485473 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:10.485599 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:10.487156 (+ 1557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:10.487163 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:10.487188 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:17:10.488853 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:10.488866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.490487 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.490491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:10.492480 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:10.492494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.494459 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.494469 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:10.494652 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.494707 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 13:17:10.494794 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 13:17:10.494911 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16253,"mutex_wait_us":322,"thread_start_us":10436,"threads_started":1}
I20260619 13:17:10.495467 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.480878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.0.97.3:38541)
0619 13:17:10.486720 (+ 5842us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:10.486729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:10.487279 (+ 550us) server_negotiation.cc:407] Connection header received
0619 13:17:10.487369 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:10.487376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:10.487566 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:10.487721 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:17:10.488592 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.490050 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.492568 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.493450 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.493564 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.493788 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:10.494034 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 13:17:10.494190 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5649,"thread_start_us":5473,"threads_started":1}
I20260619 13:17:10.495999 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.469064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:47236)
0619 13:17:10.479198 (+ 10134us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:10.479251 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:10.479479 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:10.487667 (+ 8188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:10.487671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:10.487695 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:17:10.488519 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:10.488530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.489886 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.489891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:10.492411 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:10.492420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.494485 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.494490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:10.494666 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.495077 (+ 411us) client_negotiation.cc:769] Sending connection context
0619 13:17:10.495534 (+ 457us) client_negotiation.cc:240] Negotiation successful
0619 13:17:10.495655 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9823,"mutex_wait_us":583,"thread_start_us":17475,"threads_started":1}
I20260619 13:17:10.496049 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.469416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.0.97.2:37467)
0619 13:17:10.486253 (+ 16837us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:10.486263 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:10.486294 (+ 31us) server_negotiation.cc:407] Connection header received
0619 13:17:10.486397 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:10.486405 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:10.486654 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:10.486862 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:17:10.489028 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.490367 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.492607 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.494008 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.494114 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.494300 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:10.495597 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0619 13:17:10.495756 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16657,"thread_start_us":16554,"threads_started":1}
I20260619 13:17:10.496783 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.469115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.0.97.1:44927)
0619 13:17:10.487068 (+ 17953us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:10.487076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:10.487103 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:17:10.487191 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:10.487198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:10.487493 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:10.487657 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:17:10.488650 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.489737 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.492584 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.493996 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.494114 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.494333 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:10.496338 (+ 2005us) server_negotiation.cc:299] Negotiation successful
0619 13:17:10.496501 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17803,"thread_start_us":17679,"threads_started":1}
I20260619 13:17:10.498984 16268 tablet_service.cc:1467] Processing CreateTablet for tablet c702d8e0fbe4442993c06cbace349ba6 (DEFAULT_TABLE table=test-table [id=8a9cc009255940168cb822409100b529]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:17:10.499858 16135 tablet_service.cc:1467] Processing CreateTablet for tablet c702d8e0fbe4442993c06cbace349ba6 (DEFAULT_TABLE table=test-table [id=8a9cc009255940168cb822409100b529]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:17:10.500062 16001 tablet_service.cc:1467] Processing CreateTablet for tablet c702d8e0fbe4442993c06cbace349ba6 (DEFAULT_TABLE table=test-table [id=8a9cc009255940168cb822409100b529]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:17:10.500766 16268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c702d8e0fbe4442993c06cbace349ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:17:10.501060 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c702d8e0fbe4442993c06cbace349ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:17:10.501348 16001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c702d8e0fbe4442993c06cbace349ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:17:10.507601 16066 heartbeater.cc:503] Master 127.0.97.62:34051 was elected leader, sending a full tablet report...
I20260619 13:17:10.512194 16351 tablet_bootstrap.cc:492] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea: Bootstrap starting.
I20260619 13:17:10.514675 16352 tablet_bootstrap.cc:492] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06: Bootstrap starting.
I20260619 13:17:10.515581 16351 tablet_bootstrap.cc:654] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea: Neither blocks nor log segments found. Creating new log.
I20260619 13:17:10.516069 16353 tablet_bootstrap.cc:492] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669: Bootstrap starting.
I20260619 13:17:10.516999 16351 log.cc:824] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea: Log is configured to *not* fsync() on all Append() calls
I20260619 13:17:10.518817 16352 tablet_bootstrap.cc:654] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06: Neither blocks nor log segments found. Creating new log.
I20260619 13:17:10.519208 16353 tablet_bootstrap.cc:654] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669: Neither blocks nor log segments found. Creating new log.
I20260619 13:17:10.520017 16352 log.cc:824] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06: Log is configured to *not* fsync() on all Append() calls
I20260619 13:17:10.520099 16353 log.cc:824] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669: Log is configured to *not* fsync() on all Append() calls
I20260619 13:17:10.520593 16351 tablet_bootstrap.cc:492] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea: No bootstrap required, opened a new log
I20260619 13:17:10.520882 16351 ts_tablet_manager.cc:1397] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 13:17:10.522971 16352 tablet_bootstrap.cc:492] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06: No bootstrap required, opened a new log
I20260619 13:17:10.522986 16353 tablet_bootstrap.cc:492] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669: No bootstrap required, opened a new log
I20260619 13:17:10.523219 16353 ts_tablet_manager.cc:1397] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260619 13:17:10.523347 16352 ts_tablet_manager.cc:1397] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260619 13:17:10.531093 16353 raft_consensus.cc:348] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } } peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } }
I20260619 13:17:10.531127 16351 raft_consensus.cc:348] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } } peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } }
I20260619 13:17:10.531433 16353 raft_consensus.cc:374] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:17:10.531520 16353 raft_consensus.cc:729] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f18f77dbd74c30a72c4d048de58669, State: Initialized, Role: FOLLOWER
I20260619 13:17:10.531543 16351 raft_consensus.cc:374] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:17:10.531634 16351 raft_consensus.cc:729] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 187b99c27439439b926caaa279a244ea, State: Initialized, Role: FOLLOWER
I20260619 13:17:10.532159 16353 consensus_queue.cc:260] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } } peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } }
I20260619 13:17:10.532202 16351 consensus_queue.cc:260] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } } peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } }
I20260619 13:17:10.533341 16200 heartbeater.cc:503] Master 127.0.97.62:34051 was elected leader, sending a full tablet report...
I20260619 13:17:10.534124 16351 ts_tablet_manager.cc:1428] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260619 13:17:10.534790 16352 raft_consensus.cc:348] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } } peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } }
I20260619 13:17:10.535110 16352 raft_consensus.cc:374] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:17:10.535207 16352 raft_consensus.cc:729] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ca826eca75744778b8e2d48a188dc06, State: Initialized, Role: FOLLOWER
I20260619 13:17:10.535710 16352 consensus_queue.cc:260] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } } peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } }
I20260619 13:17:10.537261 16352 ts_tablet_manager.cc:1428] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06: Time spent starting tablet: real 0.014s user 0.006s sys 0.005s
I20260619 13:17:10.538369 16333 heartbeater.cc:503] Master 127.0.97.62:34051 was elected leader, sending a full tablet report...
I20260619 13:17:10.538986 16353 ts_tablet_manager.cc:1428] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669: Time spent starting tablet: real 0.016s user 0.006s sys 0.010s
I20260619 13:17:10.638196 16359 raft_consensus.cc:482] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:17:10.638479 16359 raft_consensus.cc:504] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } } peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } }
I20260619 13:17:10.640056 16359 leader_election.cc:288] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 187b99c27439439b926caaa279a244ea (127.0.97.2:37467), 27f18f77dbd74c30a72c4d048de58669 (127.0.97.3:38541)
I20260619 13:17:10.646409 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.640580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36777 (local address 127.0.97.2:37467)
0619 13:17:10.640785 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:10.640790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:10.641104 (+ 314us) server_negotiation.cc:407] Connection header received
0619 13:17:10.641614 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:10.641620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:10.641680 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:10.641755 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:17:10.642305 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.643003 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.644741 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.645499 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.645614 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.645709 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:10.646019 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 13:17:10.646143 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:17:10.646567 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.641287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48599 (local address 127.0.97.3:38541)
0619 13:17:10.641482 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:10.641486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:10.641874 (+ 388us) server_negotiation.cc:407] Connection header received
0619 13:17:10.642059 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:10.642062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:10.642118 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:10.642188 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:17:10.642649 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.643353 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.644984 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.645720 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.645825 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.645945 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:10.646214 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 13:17:10.646347 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:17:10.646586 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.640350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37467 (local address 127.0.97.1:36777)
0619 13:17:10.641000 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:10.641019 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:10.641239 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:10.641791 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:10.641796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:10.641821 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:17:10.642169 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:10.642178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.643128 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.643133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:10.644585 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:10.644601 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.645701 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.645707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:10.645832 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.645902 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 13:17:10.645985 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 13:17:10.646110 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":296,"threads_started":1}
I20260619 13:17:10.646951 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.641070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38541 (local address 127.0.97.1:48599)
0619 13:17:10.641763 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:10.641776 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:10.641927 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:10.642216 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:10.642220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:10.642242 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:17:10.642515 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:10.642523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.643486 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.643490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:10.644837 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:10.644849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.645946 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.645952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:10.646078 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.646101 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:17:10.646175 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 13:17:10.646291 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":2,"thread_start_us":390,"threads_started":1}
I20260619 13:17:10.647617 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702d8e0fbe4442993c06cbace349ba6" candidate_uuid: "4ca826eca75744778b8e2d48a188dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "187b99c27439439b926caaa279a244ea" is_pre_election: true
I20260619 13:17:10.647920 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702d8e0fbe4442993c06cbace349ba6" candidate_uuid: "4ca826eca75744778b8e2d48a188dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f18f77dbd74c30a72c4d048de58669" is_pre_election: true
I20260619 13:17:10.648186 16155 raft_consensus.cc:2454] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ca826eca75744778b8e2d48a188dc06 in term 0.
W20260619 13:17:10.648223 16334 tablet.cc:2369] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:17:10.648473 16288 raft_consensus.cc:2454] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ca826eca75744778b8e2d48a188dc06 in term 0.
I20260619 13:17:10.648922 15954 leader_election.cc:302] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 187b99c27439439b926caaa279a244ea, 4ca826eca75744778b8e2d48a188dc06; no voters:
I20260619 13:17:10.649310 16359 raft_consensus.cc:2781] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:17:10.649462 16359 raft_consensus.cc:482] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:17:10.649567 16359 raft_consensus.cc:3037] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:17:10.651713 16359 raft_consensus.cc:504] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } } peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } }
I20260619 13:17:10.652264 16359 leader_election.cc:288] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [CANDIDATE]: Term 1 election: Requested vote from peers 187b99c27439439b926caaa279a244ea (127.0.97.2:37467), 27f18f77dbd74c30a72c4d048de58669 (127.0.97.3:38541)
I20260619 13:17:10.652632 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702d8e0fbe4442993c06cbace349ba6" candidate_uuid: "4ca826eca75744778b8e2d48a188dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "187b99c27439439b926caaa279a244ea"
I20260619 13:17:10.652697 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c702d8e0fbe4442993c06cbace349ba6" candidate_uuid: "4ca826eca75744778b8e2d48a188dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f18f77dbd74c30a72c4d048de58669"
I20260619 13:17:10.652799 16155 raft_consensus.cc:3037] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:17:10.652877 16288 raft_consensus.cc:3037] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:17:10.655089 16155 raft_consensus.cc:2454] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ca826eca75744778b8e2d48a188dc06 in term 1.
I20260619 13:17:10.655135 16288 raft_consensus.cc:2454] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ca826eca75744778b8e2d48a188dc06 in term 1.
I20260619 13:17:10.655438 15954 leader_election.cc:302] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 187b99c27439439b926caaa279a244ea, 4ca826eca75744778b8e2d48a188dc06; no voters:
I20260619 13:17:10.655712 16359 raft_consensus.cc:2781] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:17:10.656103 16359 raft_consensus.cc:686] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [term 1 LEADER]: Becoming Leader. State: Replica: 4ca826eca75744778b8e2d48a188dc06, State: Running, Role: LEADER
I20260619 13:17:10.656548 16359 consensus_queue.cc:237] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } } peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } }
I20260619 13:17:10.660858 15876 catalog_manager.cc:5495] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ca826eca75744778b8e2d48a188dc06 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4ca826eca75744778b8e2d48a188dc06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } health_report { overall_health: UNKNOWN } } }
W20260619 13:17:10.679399 16201 tablet.cc:2369] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:17:10.689055 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.684344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.0.97.1:44927)
0619 13:17:10.684537 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:10.684542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:10.684564 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:17:10.684787 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:10.684791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:10.684869 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:10.684984 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:17:10.685267 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.686007 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.687278 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.687675 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.687758 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.688145 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:17:10.688618 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:17:10.688717 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:10.688757 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 13:17:10.688817 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:17:10.696916 16288 raft_consensus.cc:1261] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669 [term 1 FOLLOWER]: Refusing update from remote peer 4ca826eca75744778b8e2d48a188dc06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:17:10.696916 16155 raft_consensus.cc:1261] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea [term 1 FOLLOWER]: Refusing update from remote peer 4ca826eca75744778b8e2d48a188dc06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:17:10.697815 16362 consensus_queue.cc:1035] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:17:10.698199 16359 consensus_queue.cc:1035] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:17:10.708323 16365 mvcc.cc:205] Tried to move back new op lower bound from 7298560125724303360 to 7298560125572411392. Current Snapshot: MvccSnapshot[applied={T|T < 7298560125724303360}]
I20260619 13:17:10.709026 16367 mvcc.cc:205] Tried to move back new op lower bound from 7298560125724303360 to 7298560125572411392. Current Snapshot: MvccSnapshot[applied={T|T < 7298560125724303360}]
I20260619 13:17:10.720842 15876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:17:10.721182 15876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:17:10.724752 15876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Sending DeleteTablet for 3 replicas of tablet c702d8e0fbe4442993c06cbace349ba6
I20260619 13:17:10.725734 16001 tablet_service.cc:1514] Processing DeleteTablet for tablet c702d8e0fbe4442993c06cbace349ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:17:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47236
I20260619 13:17:10.726167 16268 tablet_service.cc:1514] Processing DeleteTablet for tablet c702d8e0fbe4442993c06cbace349ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:17:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37226
I20260619 13:17:10.726215 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet c702d8e0fbe4442993c06cbace349ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:17:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42828
I20260619 13:17:10.727070 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 15938
I20260619 13:17:10.737362 16376 tablet_replica.cc:330] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669: stopping tablet replica
I20260619 13:17:10.737821 16376 raft_consensus.cc:2229] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:17:10.738348 16376 raft_consensus.cc:2258] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:17:10.740573 16377 tablet_replica.cc:330] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea: stopping tablet replica
I20260619 13:17:10.740844 16376 ts_tablet_manager.cc:1905] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:17:10.740969 16377 raft_consensus.cc:2229] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:17:10.741437 16377 raft_consensus.cc:2258] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:17:10.747931 16376 ts_tablet_manager.cc:1918] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:17:10.748167 16376 log.cc:1196] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/c702d8e0fbe4442993c06cbace349ba6
I20260619 13:17:10.748749 16376 ts_tablet_manager.cc:1939] T c702d8e0fbe4442993c06cbace349ba6 P 27f18f77dbd74c30a72c4d048de58669: Deleting consensus metadata
I20260619 13:17:10.746764 16377 ts_tablet_manager.cc:1905] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:17:10.750643 15861 catalog_manager.cc:4841] TS 27f18f77dbd74c30a72c4d048de58669 (127.0.97.3:38541): tablet c702d8e0fbe4442993c06cbace349ba6 (table test-table [id=8a9cc009255940168cb822409100b529]) successfully deleted
I20260619 13:17:10.756125 16375 tablet_replica.cc:330] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06: stopping tablet replica
I20260619 13:17:10.756620 16375 raft_consensus.cc:2229] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:17:10.757328 16375 raft_consensus.cc:2258] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:17:10.757458 16377 ts_tablet_manager.cc:1918] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:17:10.757619 16377 log.cc:1196] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/c702d8e0fbe4442993c06cbace349ba6
I20260619 13:17:10.758123 16377 ts_tablet_manager.cc:1939] T c702d8e0fbe4442993c06cbace349ba6 P 187b99c27439439b926caaa279a244ea: Deleting consensus metadata
Jun 19 13:17:10 dist-test-slave-mkt3 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:17:10.759737 16375 ts_tablet_manager.cc:1905] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:17:10.759927 15861 catalog_manager.cc:4841] TS 187b99c27439439b926caaa279a244ea (127.0.97.2:37467): tablet c702d8e0fbe4442993c06cbace349ba6 (table test-table [id=8a9cc009255940168cb822409100b529]) successfully deleted
I20260619 13:17:10.766353 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.728286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.0.97.1:44927)
0619 13:17:10.728471 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:10.728477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:10.742306 (+ 13829us) server_negotiation.cc:407] Connection header received
0619 13:17:10.742626 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:10.742633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:10.742719 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:10.742794 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:10.746941 (+ 4147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.748277 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.753991 (+ 5714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.754735 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.754840 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.760599 (+ 5759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:10.760610 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:10.760622 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:10.760665 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:10.763605 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:10.764275 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:10.764281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:10.764283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:10.764457 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:10.764820 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:10.764824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:10.764826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:10.765152 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:10.765473 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:10.765739 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 13:17:10.765995 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:17:10.766969 16061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:17:10.862584 16375 ts_tablet_manager.cc:1918] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:17:10.862883 16375 log.cc:1196] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/c702d8e0fbe4442993c06cbace349ba6
I20260619 13:17:10.863267 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 15938
I20260619 13:17:10.877965 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 16072
W20260619 13:17:10.878078 15864 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44927 (error 108)
I20260619 13:17:10.879715 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.879311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:47260)
0619 13:17:10.879482 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:10.879578 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 13:17:10.879928 15864 catalog_manager.cc:4584] TS 4ca826eca75744778b8e2d48a188dc06 (127.0.97.1:44927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c702d8e0fbe4442993c06cbace349ba6: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
Jun 19 13:17:10 dist-test-slave-mkt3 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:17:10.893468 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.878654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42838 (local address 127.0.97.2:37467)
0619 13:17:10.879006 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:10.879012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:10.879033 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:17:10.879102 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:10.879107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:10.879198 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:10.879286 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:10.880686 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.881417 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.882872 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.883317 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:10.883416 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:10.887511 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:10.887528 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:10.887548 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:10.887591 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:10.890952 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:10.891575 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:10.891585 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:10.891589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:10.891731 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:10.892036 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:10.892041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:10.892044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:10.892499 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:10.892786 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:10.892929 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 13:17:10.893134 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260619 13:17:10.894156 16195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:17:10.933369 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.932878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:47274)
0619 13:17:10.933070 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:10.933164 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 13:17:10.934422 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.934020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:47276)
0619 13:17:10.934191 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:10.934269 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 13:17:10.982441 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 16072
I20260619 13:17:10.994889 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.994428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:47280)
0619 13:17:10.994630 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:10.994717 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 13:17:10.996043 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.995661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:47284)
0619 13:17:10.995839 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:10.995923 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 13:17:10.996081 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 16205
Jun 19 13:17:11 dist-test-slave-mkt3 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:17:11.010792 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:10.996828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.0.97.3:38541)
0619 13:17:10.997030 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:10.997035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:10.997053 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:17:10.997107 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:10.997111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:10.997201 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:10.997285 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:10.998570 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:10.999291 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:11.000675 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:11.001082 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:11.001202 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:11.005578 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:11.005593 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:11.005610 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:11.005644 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:11.008321 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:11.008923 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:11.008930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:11.008932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:11.009068 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:11.009366 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:11.009369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:11.009370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:11.009707 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:11.009957 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:11.010307 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 13:17:11.010488 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 13:17:11.011449 16328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:17:11.076642 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:11.076208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:47286)
0619 13:17:11.076397 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:11.076499 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 13:17:11.077713 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:11.077333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:47298)
0619 13:17:11.077521 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:11.077590 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 13:17:11.099004 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 16205
I20260619 13:17:11.111872 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 15845
I20260619 13:17:11.113005 15907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:17:11.219834 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 15845
I20260619 13:17:31.238785 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34051
--webserver_interface=127.0.97.62
--webserver_port=43983
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:17:31.555768 16384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:17:31.556044 16384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:17:31.556180 16384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:17:31.563117 16384 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 13:17:31.563215 16384 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 13:17:31.563280 16384 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 13:17:31.563354 16384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:17:31.563424 16384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:17:31.563514 16384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:17:31.563571 16384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:17:31.563642 16384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:17:31.573119 16384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:17:31.574357 16384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:17:31.575590 16384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:17:31.596393 16390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:31.596418 16391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:31.596683 16393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:17:31.597236 16384 server_base.cc:1034] running on GCE node
I20260619 13:17:31.598094 16384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:17:31.599628 16384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:17:31.600844 16384 hybrid_clock.cc:648] HybridClock initialized: now 1781875051600786 us; error 34 us; skew 500 ppm
I20260619 13:17:31.601208 16384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:17:31 dist-test-slave-mkt3 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:17:31.604602 16384 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:17:31.606155 16384 webserver.cc:460] Webserver started at http://127.0.97.62:43983/ using document root <none> and password file <none>
I20260619 13:17:31.606895 16384 fs_manager.cc:359] Metadata directory not provided
I20260619 13:17:31.607010 16384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:17:31.612618 16384 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:17:31.615370 16400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:31.616590 16384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:17:31.616753 16384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "69f8e3f19c4f466cb939b9f878d6d375"
format_stamp: "Formatted at 2026-06-19 13:17:08 on dist-test-slave-mkt3"
server_key: "9ce14709b748722ec628c2c04da2862f"
server_key_iv: "dfd6dcd604480331bc94a94640dd1c75"
server_key_version: "encryptionkey@0"
I20260619 13:17:31.617734 16384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:17:31.637961 16384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:17:31.641825 16384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:17:31.642117 16384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:17:31.658910 16384 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34051
I20260619 13:17:31.658948 16451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34051 every 8 connection(s)
I20260619 13:17:31.660301 16384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:17:31.666992 16452 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:17:31.669109 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 16384
I20260619 13:17:31.670074 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44927
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34051
--webserver_port=42301
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:17:31.670761 16452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Bootstrap starting.
I20260619 13:17:31.691628 16452 log.cc:824] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Log is configured to *not* fsync() on all Append() calls
I20260619 13:17:31.703354 16452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:17:31.703851 16452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Bootstrap complete.
I20260619 13:17:31.712471 16452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 13:17:31.713245 16452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f8e3f19c4f466cb939b9f878d6d375, State: Initialized, Role: FOLLOWER
I20260619 13:17:31.713817 16452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 13:17:31.714071 16452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:17:31.714182 16452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:17:31.714334 16452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 1 FOLLOWER]: Advancing to term 2
I20260619 13:17:31.717973 16452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 13:17:31.718446 16452 leader_election.cc:302] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69f8e3f19c4f466cb939b9f878d6d375; no voters:
I20260619 13:17:31.719025 16452 leader_election.cc:288] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 13:17:31.719154 16456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 13:17:31.720283 16456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 2 LEADER]: Becoming Leader. State: Replica: 69f8e3f19c4f466cb939b9f878d6d375, State: Running, Role: LEADER
I20260619 13:17:31.720819 16456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 13:17:31.721117 16452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:17:31.730297 16458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69f8e3f19c4f466cb939b9f878d6d375. Latest consensus state: current_term: 2 leader_uuid: "69f8e3f19c4f466cb939b9f878d6d375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } } }
I20260619 13:17:31.730643 16458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [sys.catalog]: This master's current role is: LEADER
I20260619 13:17:31.733359 16457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "69f8e3f19c4f466cb939b9f878d6d375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } } }
I20260619 13:17:31.733682 16457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [sys.catalog]: This master's current role is: LEADER
I20260619 13:17:31.734498 16466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:17:31.742053 16466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:17:31.743443 16466 catalog_manager.cc:1259] Loaded cluster ID: fa2394cc402c44868c2f6d2043be19fb
I20260619 13:17:31.743583 16466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:17:31.747445 16466 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:17:31.748894 16466 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Loaded TSK: 0
I20260619 13:17:31.780028 16466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Generated new TSK 1
I20260619 13:17:31.780378 16466 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 13:17:31.998477 16454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:17:31.998744 16454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:17:31.998865 16454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:17:32.006135 16454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:17:32.006251 16454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:17:32.006394 16454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:17:32.016147 16454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42301
--tserver_master_addrs=127.0.97.62:34051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:17:32.017477 16454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:17:32.018805 16454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:17:32.033139 16481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:32.033102 16480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:32.033946 16483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:17:32.034595 16454 server_base.cc:1034] running on GCE node
I20260619 13:17:32.035153 16454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:17:32.036067 16454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:17:32.037267 16454 hybrid_clock.cc:648] HybridClock initialized: now 1781875052037209 us; error 52 us; skew 500 ppm
I20260619 13:17:32.037551 16454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:17:32 dist-test-slave-mkt3 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:17:32.040648 16454 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:17:32.042100 16454 webserver.cc:460] Webserver started at http://127.0.97.1:42301/ using document root <none> and password file <none>
I20260619 13:17:32.042742 16454 fs_manager.cc:359] Metadata directory not provided
I20260619 13:17:32.042865 16454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:17:32.048044 16454 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 13:17:32.050697 16490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:32.052013 16454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 13:17:32.052148 16454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "4ca826eca75744778b8e2d48a188dc06"
format_stamp: "Formatted at 2026-06-19 13:17:09 on dist-test-slave-mkt3"
server_key: "0b04202f7bdd8db35ebbbdcd19b4a21a"
server_key_iv: "f9ec8d1ddb5dcf1d218d266982690e55"
server_key_version: "encryptionkey@0"
I20260619 13:17:32.053062 16454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:17:32.071020 16454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:17:32.075098 16454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:17:32.075349 16454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:17:32.076385 16454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:17:32.078316 16497 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 13:17:32.084014 16497 ts_tablet_manager.cc:1852] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 13:17:32.084972 16497 ts_tablet_manager.cc:1905] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:17:32.091946 16497 ts_tablet_manager.cc:1918] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:17:32.092167 16497 ts_tablet_manager.cc:1939] T c702d8e0fbe4442993c06cbace349ba6 P 4ca826eca75744778b8e2d48a188dc06: Deleting consensus metadata
I20260619 13:17:32.092780 16454 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 13:17:32.092902 16454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.001s
I20260619 13:17:32.093016 16454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:17:32.093091 16454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:32.122949 16454 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44927
I20260619 13:17:32.123004 16603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44927 every 8 connection(s)
I20260619 13:17:32.124205 16454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:17:32.134666 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 16454
I20260619 13:17:32.135294 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37467
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34051
--webserver_port=36961
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:17:32 dist-test-slave-mkt3 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:17:32.147869 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:32.127000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56423 (local address 127.0.97.62:34051)
0619 13:17:32.127530 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:32.127542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:32.128231 (+ 689us) server_negotiation.cc:407] Connection header received
0619 13:17:32.130381 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:32.130395 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:32.130734 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:32.131057 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:32.132555 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:32.133952 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:32.135728 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:32.136689 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:32.136854 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:32.140224 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:32.140234 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:32.140253 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:32.140299 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:32.143671 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:32.144567 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:32.144574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:32.144579 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:32.144748 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:32.145182 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:32.145187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:32.145189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:32.145670 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:32.145822 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:32.146656 (+ 834us) server_negotiation.cc:299] Negotiation successful
0619 13:17:32.146970 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":229,"threads_started":1}
I20260619 13:17:32.148293 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:32.127297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:56423)
0619 13:17:32.128036 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:32.128079 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:32.129796 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:32.131219 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:32.131231 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:32.131845 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:17:32.132367 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:32.132384 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:32.134100 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:32.134104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:32.135588 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:32.135610 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:32.136888 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:32.136893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:32.137069 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:32.137726 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:17:32.137749 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:17:32.139961 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:17:32.143821 (+ 3860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:17:32.143830 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:17:32.143846 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:17:32.144425 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:17:32.144843 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:17:32.144847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:17:32.144851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:17:32.145060 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:17:32.145799 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:17:32.145805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:17:32.146202 (+ 397us) client_negotiation.cc:769] Sending connection context
0619 13:17:32.146494 (+ 292us) client_negotiation.cc:240] Negotiation successful
0619 13:17:32.146963 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":295,"threads_started":1}
I20260619 13:17:32.151086 16604 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34051
I20260619 13:17:32.151412 16604 heartbeater.cc:464] Registering TS with master...
I20260619 13:17:32.152081 16604 heartbeater.cc:511] Master 127.0.97.62:34051 requested a full tablet report, sending...
I20260619 13:17:32.154206 16417 ts_manager.cc:194] Registered new tserver with Master: 4ca826eca75744778b8e2d48a188dc06 (127.0.97.1:44927)
I20260619 13:17:32.155665 16417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56423
W20260619 13:17:32.445801 16608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:17:32.446111 16608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:17:32.446239 16608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:17:32.453337 16608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:17:32.453439 16608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:17:32.453583 16608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:17:32.463420 16608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36961
--tserver_master_addrs=127.0.97.62:34051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:17:32.464651 16608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:17:32.465971 16608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:17:32.478046 16614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:32.478078 16616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:32.478128 16613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:17:32.478900 16608 server_base.cc:1034] running on GCE node
I20260619 13:17:32.479420 16608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:17:32.480284 16608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:17:32.481489 16608 hybrid_clock.cc:648] HybridClock initialized: now 1781875052481439 us; error 43 us; skew 500 ppm
I20260619 13:17:32.481779 16608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:17:32 dist-test-slave-mkt3 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:17:32.484622 16608 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:17:32.486025 16608 webserver.cc:460] Webserver started at http://127.0.97.2:36961/ using document root <none> and password file <none>
I20260619 13:17:32.486668 16608 fs_manager.cc:359] Metadata directory not provided
I20260619 13:17:32.486799 16608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:17:32.491806 16608 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 13:17:32.494457 16623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:32.495702 16608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:17:32.495842 16608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "187b99c27439439b926caaa279a244ea"
format_stamp: "Formatted at 2026-06-19 13:17:09 on dist-test-slave-mkt3"
server_key: "dd6d0c1dbbeacc123f804406997c1ddd"
server_key_iv: "0df1b8017ca7ed6d49531e6aadcc9e88"
server_key_version: "encryptionkey@0"
I20260619 13:17:32.496734 16608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:17:32.511771 16608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:17:32.515646 16608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:17:32.515884 16608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:17:32.516798 16608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:17:32.518093 16608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:17:32.518193 16608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:32.518297 16608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:17:32.518369 16608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:32.548069 16608 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37467
I20260619 13:17:32.548135 16735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37467 every 8 connection(s)
I20260619 13:17:32.549437 16608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:17:32.559116 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 16608
I20260619 13:17:32.559800 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38541
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34051
--webserver_port=45479
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 13:17:32 dist-test-slave-mkt3 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:17:32.568105 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:32.551835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53073 (local address 127.0.97.62:34051)
0619 13:17:32.551998 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:32.552002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:32.553063 (+ 1061us) server_negotiation.cc:407] Connection header received
0619 13:17:32.554890 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:32.554896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:32.554980 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:32.555066 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:32.556525 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:32.557233 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:32.558798 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:32.559213 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:32.559326 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:32.563038 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:32.563043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:32.563047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:32.563079 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:32.565302 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:32.565983 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:32.565987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:32.565990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:32.566104 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:32.566487 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:32.566490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:32.566492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:32.566765 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:32.566890 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:32.567529 (+ 639us) server_negotiation.cc:299] Negotiation successful
0619 13:17:32.567765 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 13:17:32.569012 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:32.552236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:53073)
0619 13:17:32.552915 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:32.552952 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:32.554626 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:32.555301 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:32.555309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:32.555820 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:17:32.556363 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:32.556380 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:32.557380 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:32.557384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:32.558661 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:32.558672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:32.559430 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:32.559438 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:32.559716 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:32.560407 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:17:32.560435 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:17:32.562829 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:17:32.565431 (+ 2602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:17:32.565440 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:17:32.565456 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:17:32.565871 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:17:32.566202 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:17:32.566206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:17:32.566210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:17:32.566382 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:17:32.566879 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:17:32.566887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:17:32.567203 (+ 316us) client_negotiation.cc:769] Sending connection context
0619 13:17:32.567416 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 13:17:32.567906 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":248,"threads_started":1}
I20260619 13:17:32.570369 16736 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34051
I20260619 13:17:32.570657 16736 heartbeater.cc:464] Registering TS with master...
I20260619 13:17:32.571319 16736 heartbeater.cc:511] Master 127.0.97.62:34051 requested a full tablet report, sending...
I20260619 13:17:32.572659 16417 ts_manager.cc:194] Registered new tserver with Master: 187b99c27439439b926caaa279a244ea (127.0.97.2:37467)
I20260619 13:17:32.573678 16417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53073
W20260619 13:17:32.869341 16739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:17:32.869586 16739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:17:32.869676 16739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:17:32.876717 16739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:17:32.876819 16739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:17:32.876928 16739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:17:32.886209 16739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45479
--tserver_master_addrs=127.0.97.62:34051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:17:32.887390 16739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:17:32.888564 16739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:17:32.900925 16744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:32.900959 16747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:32.900851 16745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:17:32.901660 16739 server_base.cc:1034] running on GCE node
I20260619 13:17:32.902264 16739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:17:32.903168 16739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:17:32.904376 16739 hybrid_clock.cc:648] HybridClock initialized: now 1781875052904314 us; error 55 us; skew 500 ppm
I20260619 13:17:32.904675 16739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:17:32 dist-test-slave-mkt3 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:17:32.907650 16739 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:17:32.909096 16739 webserver.cc:460] Webserver started at http://127.0.97.3:45479/ using document root <none> and password file <none>
I20260619 13:17:32.909729 16739 fs_manager.cc:359] Metadata directory not provided
I20260619 13:17:32.909873 16739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:17:32.915019 16739 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:17:32.917634 16754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:32.918956 16739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:17:32.919106 16739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "27f18f77dbd74c30a72c4d048de58669"
format_stamp: "Formatted at 2026-06-19 13:17:10 on dist-test-slave-mkt3"
server_key: "771c5a9accb251e73fcfc85a51ca907e"
server_key_iv: "5c44bf437847cb2e438d3f8dfb6dc2f0"
server_key_version: "encryptionkey@0"
I20260619 13:17:32.920034 16739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:17:32.938694 16739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:17:32.942369 16739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:17:32.942612 16739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:17:32.943487 16739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:17:32.944662 16739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:17:32.944752 16739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:32.944859 16739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:17:32.944928 16739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:32.977769 16739 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38541
I20260619 13:17:32.977816 16866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38541 every 8 connection(s)
I20260619 13:17:32.979156 16739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:17:32.981984 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 16739
Jun 19 13:17:32 dist-test-slave-mkt3 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:17:32.999531 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:32.982950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.0.97.62:34051)
0619 13:17:32.986556 (+ 3606us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:32.986560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:32.986998 (+ 438us) server_negotiation.cc:407] Connection header received
0619 13:17:32.987167 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:32.987171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:32.987226 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:32.987315 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:32.988779 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:32.989542 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:32.990903 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:32.991334 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:32.991436 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:32.993955 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:32.993960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:32.993964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:32.993993 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:32.996766 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:32.997512 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:32.997518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:32.997521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:32.997651 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:32.998034 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:32.998037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:32.998039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:32.998353 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:32.998603 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:32.998909 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 13:17:32.999109 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3515,"thread_start_us":3380,"threads_started":1}
I20260619 13:17:33.000492 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:32.981612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59501 (local address 127.0.97.62:34051)
0619 13:17:32.981777 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:32.981781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:32.983386 (+ 1605us) server_negotiation.cc:407] Connection header received
0619 13:17:32.986420 (+ 3034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:32.986427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:32.986525 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:32.986601 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:32.988299 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:32.989057 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:32.990564 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:32.990989 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:32.991095 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:32.994515 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:32.994520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:32.994523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:32.994560 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:32.997214 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:32.997998 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:32.998004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:32.998007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:32.998123 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:32.998582 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:32.998587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:32.998591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:32.998906 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:32.999031 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:32.999983 (+ 952us) server_negotiation.cc:299] Negotiation successful
0619 13:17:33.000171 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:17:33.001726 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:32.982288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:59501)
0619 13:17:32.983165 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:32.983255 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:32.985945 (+ 2690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:32.987073 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:32.987082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:32.987581 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:17:32.988124 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:32.988141 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:32.989214 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:32.989218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:32.990426 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:32.990438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:32.991201 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:32.991209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:32.991449 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:32.992066 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:17:32.992096 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:17:32.994319 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:17:32.997330 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:17:32.997337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:17:32.997348 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:17:32.997891 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:17:32.998221 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:17:32.998226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:17:32.998229 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:17:32.998469 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:17:32.999024 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:17:32.999032 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:17:32.999426 (+ 394us) client_negotiation.cc:769] Sending connection context
0619 13:17:32.999843 (+ 417us) client_negotiation.cc:240] Negotiation successful
0619 13:17:33.000486 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":345,"threads_started":1}
I20260619 13:17:33.003021 16867 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34051
I20260619 13:17:33.003307 16867 heartbeater.cc:464] Registering TS with master...
I20260619 13:17:33.003988 16867 heartbeater.cc:511] Master 127.0.97.62:34051 requested a full tablet report, sending...
I20260619 13:17:33.005282 16417 ts_manager.cc:194] Registered new tserver with Master: 27f18f77dbd74c30a72c4d048de58669 (127.0.97.3:38541)
I20260619 13:17:33.006088 16417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59501
I20260619 13:17:33.014405 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:17:33.020480 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.015212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.0.97.62:34051)
0619 13:17:33.015353 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:33.015357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:33.015788 (+ 431us) server_negotiation.cc:407] Connection header received
0619 13:17:33.015955 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:33.015959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:33.016088 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:33.016230 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:17:33.016611 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.017375 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.018845 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.019281 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.019375 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.019774 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:17:33.019977 (+ 203us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 13:17:33.020270 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58300: token expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 13:17:33.020614 16606 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58300: token expired
W20260619 13:17:33.021042 16337 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:34051)
I20260619 13:17:33.032613 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.021485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58310 (local address 127.0.97.62:34051)
0619 13:17:33.021621 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:33.021625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:33.021639 (+ 14us) server_negotiation.cc:407] Connection header received
0619 13:17:33.021722 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:33.021725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:33.021777 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:33.021885 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:33.023173 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.023966 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.025266 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.025665 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.025755 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.028143 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:33.028148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:33.028151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:33.028188 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:33.030525 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:33.031077 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:33.031081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:33.031083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:33.031205 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:33.031480 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:33.031483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:33.031485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:33.031755 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:33.031920 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:33.032217 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 13:17:33.032373 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 13:17:33.035815 16417 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:17:33.042742 16417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:17:33.064355 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.053493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37467 (local address 127.0.0.1:49658)
0619 13:17:33.054311 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:33.054352 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:33.054528 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:33.056123 (+ 1595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:33.056131 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:33.056164 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:17:33.056580 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:33.056594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.058851 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.058857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:33.061394 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:33.061412 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.063364 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.063371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:33.063577 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.063635 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 13:17:33.063752 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 13:17:33.063946 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":1160,"threads_started":1}
I20260619 13:17:33.070528 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.053626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49658 (local address 127.0.97.2:37467)
0619 13:17:33.055500 (+ 1874us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:33.055508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:33.055540 (+ 32us) server_negotiation.cc:407] Connection header received
0619 13:17:33.055635 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:33.055643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:33.055891 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:33.056799 (+ 908us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:17:33.057086 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.058705 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.061619 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.063095 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.069720 (+ 6625us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.069993 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:33.070060 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 13:17:33.070224 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1736,"thread_start_us":1605,"threads_started":1}
I20260619 13:17:33.072005 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.054045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:57968)
0619 13:17:33.064595 (+ 10550us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:33.064615 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:33.064719 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:33.065728 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:33.065734 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:33.065760 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:17:33.065956 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:33.065966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.067812 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.067818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:33.069628 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:33.069642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.071282 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.071287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:33.071480 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.071522 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 13:17:33.071604 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 13:17:33.071735 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10453,"thread_start_us":15408,"threads_started":1}
I20260619 13:17:33.072719 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.054455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.0.97.1:44927)
0619 13:17:33.055108 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:33.055117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:33.064943 (+ 9826us) server_negotiation.cc:407] Connection header received
0619 13:17:33.065054 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:33.065062 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:33.065334 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:33.065519 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:17:33.066121 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.067631 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.069793 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.071121 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.071245 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.071424 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:33.072252 (+ 828us) server_negotiation.cc:299] Negotiation successful
0619 13:17:33.072406 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":1663,"threads_started":1}
I20260619 13:17:33.074241 16671 tablet_service.cc:1467] Processing CreateTablet for tablet 713208afe6854e9b86169f974aa9bbab (DEFAULT_TABLE table=test-table [id=3c7618223af6437bb9391e434dcefbe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:17:33.074674 16539 tablet_service.cc:1467] Processing CreateTablet for tablet 713208afe6854e9b86169f974aa9bbab (DEFAULT_TABLE table=test-table [id=3c7618223af6437bb9391e434dcefbe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:17:33.075362 16671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 713208afe6854e9b86169f974aa9bbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:17:33.075912 16539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 713208afe6854e9b86169f974aa9bbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:17:33.077694 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.055422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38541 (local address 127.0.0.1:48158)
0619 13:17:33.069594 (+ 14172us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:33.069618 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:33.070282 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:33.070845 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:33.070849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:33.070873 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:17:33.071138 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:33.071147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.073183 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.073188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:33.074971 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:33.074985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.076952 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.076958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:33.077142 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.077175 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:17:33.077261 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 13:17:33.077385 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14042,"thread_start_us":14544,"threads_started":1}
I20260619 13:17:33.078562 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.055914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.0.97.3:38541)
0619 13:17:33.069093 (+ 13179us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:33.069101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:33.069722 (+ 621us) server_negotiation.cc:407] Connection header received
0619 13:17:33.070443 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:33.070449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:33.070629 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:33.070800 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:17:33.071334 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.073055 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.075164 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.076480 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.076598 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.076808 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:33.078085 (+ 1277us) server_negotiation.cc:299] Negotiation successful
0619 13:17:33.078257 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13007,"thread_start_us":13199,"threads_started":1}
I20260619 13:17:33.083551 16883 tablet_bootstrap.cc:492] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06: Bootstrap starting.
I20260619 13:17:33.086844 16883 tablet_bootstrap.cc:654] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06: Neither blocks nor log segments found. Creating new log.
I20260619 13:17:33.086602 16802 tablet_service.cc:1467] Processing CreateTablet for tablet 713208afe6854e9b86169f974aa9bbab (DEFAULT_TABLE table=test-table [id=3c7618223af6437bb9391e434dcefbe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:17:33.087392 16884 tablet_bootstrap.cc:492] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea: Bootstrap starting.
I20260619 13:17:33.088128 16802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 713208afe6854e9b86169f974aa9bbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:17:33.088119 16883 log.cc:824] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06: Log is configured to *not* fsync() on all Append() calls
I20260619 13:17:33.090508 16884 tablet_bootstrap.cc:654] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea: Neither blocks nor log segments found. Creating new log.
I20260619 13:17:33.091087 16883 tablet_bootstrap.cc:492] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06: No bootstrap required, opened a new log
I20260619 13:17:33.091341 16883 ts_tablet_manager.cc:1397] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:17:33.091701 16884 log.cc:824] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea: Log is configured to *not* fsync() on all Append() calls
I20260619 13:17:33.094542 16884 tablet_bootstrap.cc:492] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea: No bootstrap required, opened a new log
I20260619 13:17:33.094796 16884 ts_tablet_manager.cc:1397] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260619 13:17:33.099722 16887 tablet_bootstrap.cc:492] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669: Bootstrap starting.
I20260619 13:17:33.101351 16883 raft_consensus.cc:348] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } } peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } }
I20260619 13:17:33.101773 16883 raft_consensus.cc:374] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:17:33.101897 16883 raft_consensus.cc:729] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ca826eca75744778b8e2d48a188dc06, State: Initialized, Role: FOLLOWER
I20260619 13:17:33.102497 16883 consensus_queue.cc:260] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } } peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } }
I20260619 13:17:33.102777 16887 tablet_bootstrap.cc:654] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669: Neither blocks nor log segments found. Creating new log.
I20260619 13:17:33.103543 16604 heartbeater.cc:503] Master 127.0.97.62:34051 was elected leader, sending a full tablet report...
I20260619 13:17:33.104149 16883 ts_tablet_manager.cc:1428] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260619 13:17:33.104668 16887 log.cc:824] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669: Log is configured to *not* fsync() on all Append() calls
I20260619 13:17:33.104760 16884 raft_consensus.cc:348] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } } peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } }
I20260619 13:17:33.105221 16884 raft_consensus.cc:374] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:17:33.105453 16884 raft_consensus.cc:729] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 187b99c27439439b926caaa279a244ea, State: Initialized, Role: FOLLOWER
I20260619 13:17:33.106019 16884 consensus_queue.cc:260] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } } peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } }
I20260619 13:17:33.107069 16736 heartbeater.cc:503] Master 127.0.97.62:34051 was elected leader, sending a full tablet report...
I20260619 13:17:33.107707 16884 ts_tablet_manager.cc:1428] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
I20260619 13:17:33.107795 16887 tablet_bootstrap.cc:492] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669: No bootstrap required, opened a new log
I20260619 13:17:33.108089 16887 ts_tablet_manager.cc:1397] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 13:17:33.115736 16887 raft_consensus.cc:348] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } } peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } }
I20260619 13:17:33.116052 16887 raft_consensus.cc:374] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:17:33.116153 16887 raft_consensus.cc:729] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f18f77dbd74c30a72c4d048de58669, State: Initialized, Role: FOLLOWER
I20260619 13:17:33.116647 16887 consensus_queue.cc:260] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } } peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } }
I20260619 13:17:33.117640 16867 heartbeater.cc:503] Master 127.0.97.62:34051 was elected leader, sending a full tablet report...
I20260619 13:17:33.118242 16887 ts_tablet_manager.cc:1428] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669: Time spent starting tablet: real 0.010s user 0.004s sys 0.006s
I20260619 13:17:33.123723 16888 raft_consensus.cc:482] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:17:33.123984 16888 raft_consensus.cc:504] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } } peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } }
I20260619 13:17:33.125471 16888 leader_election.cc:288] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 187b99c27439439b926caaa279a244ea (127.0.97.2:37467), 27f18f77dbd74c30a72c4d048de58669 (127.0.97.3:38541)
W20260619 13:17:33.126281 16605 tablet.cc:2369] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:17:33.133942 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.127260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49347 (local address 127.0.97.3:38541)
0619 13:17:33.127421 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:33.127427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:33.127445 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:17:33.127577 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:33.127582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:33.127647 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:33.127706 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:17:33.128442 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.129143 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.131795 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.132991 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.133132 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.133252 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:33.133548 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 13:17:33.133665 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:17:33.134122 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.126770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38541 (local address 127.0.97.1:49347)
0619 13:17:33.127313 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:33.127329 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:33.127426 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:33.127886 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:33.127890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:33.127915 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:17:33.128299 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:33.128308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.129269 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.129275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:33.131648 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:33.131657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.133181 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.133188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:33.133398 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.133424 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 13:17:33.133509 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 13:17:33.133631 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":24,"thread_start_us":675,"threads_started":1}
I20260619 13:17:33.135000 16822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713208afe6854e9b86169f974aa9bbab" candidate_uuid: "4ca826eca75744778b8e2d48a188dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f18f77dbd74c30a72c4d048de58669" is_pre_election: true
I20260619 13:17:33.135430 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.126345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33741 (local address 127.0.97.2:37467)
0619 13:17:33.127078 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:33.127084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:33.127196 (+ 112us) server_negotiation.cc:407] Connection header received
0619 13:17:33.127265 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:33.127270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:33.127342 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:33.127421 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:17:33.128244 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.129269 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.133341 (+ 4072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.134546 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.134663 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.134751 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:33.135073 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 13:17:33.135204 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588}
I20260619 13:17:33.135491 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.125827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37467 (local address 127.0.97.1:33741)
0619 13:17:33.126506 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:33.126530 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:17:33.126663 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:17:33.127602 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:17:33.127606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:17:33.127628 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:17:33.128083 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:33.128099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.129406 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.129411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:33.131642 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:17:33.131657 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.134745 (+ 3088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.134751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:17:33.134907 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.134936 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 13:17:33.135030 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 13:17:33.135147 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":21,"thread_start_us":538,"threads_started":1}
I20260619 13:17:33.135483 16822 raft_consensus.cc:2454] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ca826eca75744778b8e2d48a188dc06 in term 0.
I20260619 13:17:33.136420 16491 leader_election.cc:302] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f18f77dbd74c30a72c4d048de58669, 4ca826eca75744778b8e2d48a188dc06; no voters:
I20260619 13:17:33.136544 16691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713208afe6854e9b86169f974aa9bbab" candidate_uuid: "4ca826eca75744778b8e2d48a188dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "187b99c27439439b926caaa279a244ea" is_pre_election: true
I20260619 13:17:33.136813 16888 raft_consensus.cc:2781] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:17:33.136957 16888 raft_consensus.cc:482] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:17:33.137048 16888 raft_consensus.cc:3037] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:17:33.137006 16691 raft_consensus.cc:2454] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ca826eca75744778b8e2d48a188dc06 in term 0.
I20260619 13:17:33.139340 16888 raft_consensus.cc:504] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } } peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } }
I20260619 13:17:33.139881 16888 leader_election.cc:288] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [CANDIDATE]: Term 1 election: Requested vote from peers 187b99c27439439b926caaa279a244ea (127.0.97.2:37467), 27f18f77dbd74c30a72c4d048de58669 (127.0.97.3:38541)
I20260619 13:17:33.140175 16691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713208afe6854e9b86169f974aa9bbab" candidate_uuid: "4ca826eca75744778b8e2d48a188dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "187b99c27439439b926caaa279a244ea"
I20260619 13:17:33.140331 16691 raft_consensus.cc:3037] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:17:33.140337 16822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "713208afe6854e9b86169f974aa9bbab" candidate_uuid: "4ca826eca75744778b8e2d48a188dc06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f18f77dbd74c30a72c4d048de58669"
I20260619 13:17:33.140528 16822 raft_consensus.cc:3037] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:17:33.142504 16691 raft_consensus.cc:2454] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ca826eca75744778b8e2d48a188dc06 in term 1.
I20260619 13:17:33.142709 16822 raft_consensus.cc:2454] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ca826eca75744778b8e2d48a188dc06 in term 1.
I20260619 13:17:33.142843 16491 leader_election.cc:302] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 187b99c27439439b926caaa279a244ea, 4ca826eca75744778b8e2d48a188dc06; no voters:
I20260619 13:17:33.143101 16888 raft_consensus.cc:2781] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:17:33.143501 16888 raft_consensus.cc:686] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [term 1 LEADER]: Becoming Leader. State: Replica: 4ca826eca75744778b8e2d48a188dc06, State: Running, Role: LEADER
I20260619 13:17:33.143914 16888 consensus_queue.cc:237] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } } peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } }
I20260619 13:17:33.147976 16417 catalog_manager.cc:5495] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ca826eca75744778b8e2d48a188dc06 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4ca826eca75744778b8e2d48a188dc06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ca826eca75744778b8e2d48a188dc06" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44927 } health_report { overall_health: HEALTHY } } }
I20260619 13:17:33.165179 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.160801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.0.97.1:44927)
0619 13:17:33.160955 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:33.160960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:33.160976 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:17:33.161102 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:33.161106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:33.161180 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:33.161297 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:17:33.161570 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.162325 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.163490 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.163870 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.163953 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.164309 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:17:33.164782 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:17:33.164885 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:33.164924 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 13:17:33.164983 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 13:17:33.172717 16691 raft_consensus.cc:1261] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea [term 1 FOLLOWER]: Refusing update from remote peer 4ca826eca75744778b8e2d48a188dc06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:17:33.172770 16822 raft_consensus.cc:1261] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669 [term 1 FOLLOWER]: Refusing update from remote peer 4ca826eca75744778b8e2d48a188dc06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:17:33.173599 16894 consensus_queue.cc:1035] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f18f77dbd74c30a72c4d048de58669" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:17:33.173983 16888 consensus_queue.cc:1035] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "187b99c27439439b926caaa279a244ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:17:33.184928 16896 mvcc.cc:205] Tried to move back new op lower bound from 7298560217786585088 to 7298560217680637952. Current Snapshot: MvccSnapshot[applied={T|T < 7298560217786585088}]
I20260619 13:17:33.197958 16417 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:17:33.198172 16417 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:17:33.201448 16417 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Sending DeleteTablet for 3 replicas of tablet 713208afe6854e9b86169f974aa9bbab
I20260619 13:17:33.202670 16539 tablet_service.cc:1514] Processing DeleteTablet for tablet 713208afe6854e9b86169f974aa9bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:17:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57968
I20260619 13:17:33.202662 16671 tablet_service.cc:1514] Processing DeleteTablet for tablet 713208afe6854e9b86169f974aa9bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:17:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49658
I20260619 13:17:33.202896 16802 tablet_service.cc:1514] Processing DeleteTablet for tablet 713208afe6854e9b86169f974aa9bbab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:17:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48158
I20260619 13:17:33.204093 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 16454
I20260619 13:17:33.214123 16908 tablet_replica.cc:330] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669: stopping tablet replica
I20260619 13:17:33.214623 16908 raft_consensus.cc:2229] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:17:33.215137 16908 raft_consensus.cc:2258] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:17:33.216054 16906 tablet_replica.cc:330] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06: stopping tablet replica
I20260619 13:17:33.216496 16906 raft_consensus.cc:2229] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:17:33.217147 16906 raft_consensus.cc:2258] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:17:33.217917 16908 ts_tablet_manager.cc:1905] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:17:33.218832 16906 ts_tablet_manager.cc:1905] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:17:33.222483 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.204933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.0.97.1:44927)
0619 13:17:33.205108 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:33.205115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:33.205242 (+ 127us) server_negotiation.cc:407] Connection header received
0619 13:17:33.205429 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:33.205434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:33.205512 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:33.205625 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:33.207502 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.208849 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.210874 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.211572 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.211693 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.215067 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:33.215080 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:33.215097 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:33.215146 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:33.219280 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:33.220067 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:33.220076 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:33.220080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:33.220262 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:33.220661 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:33.220665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:33.220669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:33.221163 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:33.221456 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:33.221872 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 13:17:33.222128 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:17:33.223115 16599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:17:33.225713 16908 ts_tablet_manager.cc:1918] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:17:33.225906 16908 log.cc:1196] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/713208afe6854e9b86169f974aa9bbab
I20260619 13:17:33.226426 16908 ts_tablet_manager.cc:1939] T 713208afe6854e9b86169f974aa9bbab P 27f18f77dbd74c30a72c4d048de58669: Deleting consensus metadata
I20260619 13:17:33.227721 16907 tablet_replica.cc:330] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea: stopping tablet replica
I20260619 13:17:33.228078 16907 raft_consensus.cc:2229] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:17:33.228394 16401 catalog_manager.cc:4841] TS 27f18f77dbd74c30a72c4d048de58669 (127.0.97.3:38541): tablet 713208afe6854e9b86169f974aa9bbab (table test-table [id=3c7618223af6437bb9391e434dcefbe5]) successfully deleted
I20260619 13:17:33.228574 16907 raft_consensus.cc:2258] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:17:33.230499 16907 ts_tablet_manager.cc:1905] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:17:33.237139 16907 ts_tablet_manager.cc:1918] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:17:33.237293 16907 log.cc:1196] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/713208afe6854e9b86169f974aa9bbab
I20260619 13:17:33.237758 16907 ts_tablet_manager.cc:1939] T 713208afe6854e9b86169f974aa9bbab P 187b99c27439439b926caaa279a244ea: Deleting consensus metadata
I20260619 13:17:33.239432 16401 catalog_manager.cc:4841] TS 187b99c27439439b926caaa279a244ea (127.0.97.2:37467): tablet 713208afe6854e9b86169f974aa9bbab (table test-table [id=3c7618223af6437bb9391e434dcefbe5]) successfully deleted
I20260619 13:17:33.325397 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 16454
W20260619 13:17:33.339905 16404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44927 (error 108)
I20260619 13:17:33.340070 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 16608
I20260619 13:17:33.341717 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.341264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:58008)
0619 13:17:33.341421 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:33.341554 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 13:17:33.341969 16404 catalog_manager.cc:4584] TS 4ca826eca75744778b8e2d48a188dc06 (127.0.97.1:44927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 713208afe6854e9b86169f974aa9bbab: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
I20260619 13:17:33.356478 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.340827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.0.97.2:37467)
0619 13:17:33.340999 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:33.341005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:33.341107 (+ 102us) server_negotiation.cc:407] Connection header received
0619 13:17:33.341303 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:33.341309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:33.341394 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:33.341503 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:33.342973 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.343691 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.344902 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.345287 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.345370 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.351510 (+ 6140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:33.351524 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:33.351538 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:33.351579 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:33.354274 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:33.354832 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:33.354839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:33.354841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:33.354955 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:33.355300 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:33.355303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:33.355305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:33.355622 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:33.355828 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:33.356025 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 13:17:33.356173 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 13:17:33.357079 16731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:17:33.395517 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.395045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:58012)
0619 13:17:33.395264 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:33.395365 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 13:17:33.396552 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.396137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:58014)
0619 13:17:33.396344 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:33.396424 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 13:17:33.444872 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 16608
I20260619 13:17:33.456980 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.456506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:58026)
0619 13:17:33.456727 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:33.456820 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 13:17:33.458094 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 16739
I20260619 13:17:33.458195 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.457705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:58028)
0619 13:17:33.457969 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:33.458051 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260619 13:17:33.470297 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.458829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.0.97.3:38541)
0619 13:17:33.458980 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 13:17:33.458985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:17:33.459002 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:17:33.459106 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:17:33.459110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:17:33.459186 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:17:33.459268 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:17:33.460468 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.461199 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.462467 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:17:33.462856 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:17:33.462943 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:17:33.465189 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:17:33.465203 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:17:33.465219 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:17:33.465249 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:17:33.467940 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:33.468523 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:33.468530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:33.468532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:33.468660 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:17:33.469036 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:17:33.469039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:17:33.469041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:17:33.469361 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:17:33.469564 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 13:17:33.469890 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 13:17:33.470053 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:17:33.470887 16862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:17:33.538710 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.538277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:58034)
0619 13:17:33.538467 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:33.538557 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 13:17:33.539825 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:33.539423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44927 (local address 127.0.0.1:58050)
0619 13:17:33.539629 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:33.539705 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 13:17:33.557329 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 16739
I20260619 13:17:33.570323 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 16384
I20260619 13:17:33.571333 16447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:17:33.683086 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 16384
I20260619 13:17:53.700618 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34051
--webserver_interface=127.0.97.62
--webserver_port=43983
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:17:54.009037 16914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:17:54.009323 16914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:17:54.009413 16914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:17:54.016523 16914 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 13:17:54.016619 16914 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 13:17:54.016655 16914 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 13:17:54.016701 16914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:17:54.016752 16914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:17:54.016819 16914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:17:54.016862 16914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:17:54.016912 16914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:17:54.026556 16914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:17:54.027725 16914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:17:54.028956 16914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:17:54.039906 16919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:54.039980 16920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:54.040388 16922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:17:54.040606 16914 server_base.cc:1034] running on GCE node
I20260619 13:17:54.041348 16914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:17:54.042714 16914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:17:54.043917 16914 hybrid_clock.cc:648] HybridClock initialized: now 1781875074043858 us; error 40 us; skew 500 ppm
I20260619 13:17:54.044219 16914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:17:54 dist-test-slave-mkt3 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:17:54.051729 16914 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:17:54.053072 16914 webserver.cc:460] Webserver started at http://127.0.97.62:43983/ using document root <none> and password file <none>
I20260619 13:17:54.053743 16914 fs_manager.cc:359] Metadata directory not provided
I20260619 13:17:54.053860 16914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:17:54.059054 16914 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:17:54.061751 16929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:54.063023 16914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:17:54.063174 16914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "69f8e3f19c4f466cb939b9f878d6d375"
format_stamp: "Formatted at 2026-06-19 13:17:08 on dist-test-slave-mkt3"
server_key: "9ce14709b748722ec628c2c04da2862f"
server_key_iv: "dfd6dcd604480331bc94a94640dd1c75"
server_key_version: "encryptionkey@0"
I20260619 13:17:54.064085 16914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:17:54.085116 16914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:17:54.089316 16914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:17:54.089570 16914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:17:54.105585 16914 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34051
I20260619 13:17:54.105634 16980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34051 every 8 connection(s)
I20260619 13:17:54.107038 16914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:17:54.110430 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 16914
I20260619 13:17:54.111348 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44927
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34051
--webserver_port=42301
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:17:54.114967 16981 sys_catalog.cc:263] Verifying existing consensus state
I20260619 13:17:54.119204 16981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Bootstrap starting.
I20260619 13:17:54.138969 16981 log.cc:824] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Log is configured to *not* fsync() on all Append() calls
I20260619 13:17:54.154524 16981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 13:17:54.155016 16981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Bootstrap complete.
I20260619 13:17:54.164184 16981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 13:17:54.165063 16981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f8e3f19c4f466cb939b9f878d6d375, State: Initialized, Role: FOLLOWER
I20260619 13:17:54.165653 16981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 13:17:54.165907 16981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:17:54.166013 16981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:17:54.166174 16981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 2 FOLLOWER]: Advancing to term 3
I20260619 13:17:54.170192 16981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 13:17:54.170632 16981 leader_election.cc:302] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69f8e3f19c4f466cb939b9f878d6d375; no voters:
I20260619 13:17:54.171228 16981 leader_election.cc:288] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 13:17:54.171370 16985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 13:17:54.173207 16981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:17:54.178270 16985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [term 3 LEADER]: Becoming Leader. State: Replica: 69f8e3f19c4f466cb939b9f878d6d375, State: Running, Role: LEADER
I20260619 13:17:54.179116 16985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } }
I20260619 13:17:54.181713 16986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "69f8e3f19c4f466cb939b9f878d6d375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } } }
I20260619 13:17:54.182080 16986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [sys.catalog]: This master's current role is: LEADER
I20260619 13:17:54.182036 16987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69f8e3f19c4f466cb939b9f878d6d375. Latest consensus state: current_term: 3 leader_uuid: "69f8e3f19c4f466cb939b9f878d6d375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69f8e3f19c4f466cb939b9f878d6d375" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34051 } } }
I20260619 13:17:54.182372 16987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375 [sys.catalog]: This master's current role is: LEADER
I20260619 13:17:54.194834 16995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:17:54.203645 16995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:17:54.205281 16995 catalog_manager.cc:1259] Loaded cluster ID: fa2394cc402c44868c2f6d2043be19fb
I20260619 13:17:54.205444 16995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:17:54.209697 16995 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:17:54.211719 16995 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Loaded TSK: 0
I20260619 13:17:54.211956 16995 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 69f8e3f19c4f466cb939b9f878d6d375: Loaded TSK: 1
F20260619 13:17:54.213172 16995 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5e719709ad google::LogMessage::Fail() at ??:0
@ 0x7f5e7197388a google::LogMessage::SendToLog() at ??:0
@ 0x7f5e719704de google::LogMessage::Flush() at ??:0
@ 0x7f5e71971a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5e756d47d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5e756c373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f5e8739f1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f5e8739e783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5e8742a2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f5e8742a0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f5e874745e9 std::function<>::operator()() at ??:0
@ 0x7f5e873a2cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f5e873a0c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5e87428610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f5e87428439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5e745bc345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5e745bf91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f5e745bf799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f5e7459aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5e778746db start_thread at ??:0
@ 0x7f5e7080a61f clone at ??:0
W20260619 13:17:54.470827 16983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:17:54.471112 16983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:17:54.471257 16983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:17:54.478552 16983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:17:54.478677 16983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:17:54.478830 16983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:17:54.488559 16983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42301
--tserver_master_addrs=127.0.97.62:34051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:17:54.489800 16983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:17:54.491081 16983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:17:54.503333 17029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:54.503333 17028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:54.503592 17031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:17:54.504120 16983 server_base.cc:1034] running on GCE node
I20260619 13:17:54.504733 16983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:17:54.505635 16983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:17:54.506863 16983 hybrid_clock.cc:648] HybridClock initialized: now 1781875074506803 us; error 56 us; skew 500 ppm
I20260619 13:17:54.507184 16983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:17:54 dist-test-slave-mkt3 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:17:54.510156 16983 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:17:54.511564 16983 webserver.cc:460] Webserver started at http://127.0.97.1:42301/ using document root <none> and password file <none>
I20260619 13:17:54.512215 16983 fs_manager.cc:359] Metadata directory not provided
I20260619 13:17:54.512334 16983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:17:54.517400 16983 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:17:54.520197 17038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:54.521427 16983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:17:54.521564 16983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "4ca826eca75744778b8e2d48a188dc06"
format_stamp: "Formatted at 2026-06-19 13:17:09 on dist-test-slave-mkt3"
server_key: "0b04202f7bdd8db35ebbbdcd19b4a21a"
server_key_iv: "f9ec8d1ddb5dcf1d218d266982690e55"
server_key_version: "encryptionkey@0"
I20260619 13:17:54.522518 16983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:17:54.556878 16983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:17:54.560962 16983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:17:54.561219 16983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:17:54.562180 16983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:17:54.563793 17045 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 13:17:54.568869 17045 ts_tablet_manager.cc:1852] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 13:17:54.569808 17045 ts_tablet_manager.cc:1905] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:17:54.576651 17045 ts_tablet_manager.cc:1918] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:17:54.576821 17045 log.cc:1196] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/713208afe6854e9b86169f974aa9bbab
I20260619 13:17:54.577261 17045 ts_tablet_manager.cc:1939] T 713208afe6854e9b86169f974aa9bbab P 4ca826eca75744778b8e2d48a188dc06: Deleting consensus metadata
I20260619 13:17:54.577580 16983 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 13:17:54.577692 16983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20260619 13:17:54.577796 16983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:17:54.577898 16983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:54.607165 16983 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44927
I20260619 13:17:54.607208 17151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44927 every 8 connection(s)
I20260619 13:17:54.608387 16983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:17:54.612915 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:54.611186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:45419)
0619 13:17:54.611888 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:54.612063 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":267,"threads_started":1}
W20260619 13:17:54.613503 17152 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34051 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34051: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
I20260619 13:17:54.614316 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:54.613964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:57725)
0619 13:17:54.614122 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:54.614213 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 13:17:54.615134 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:54.614815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:42455)
0619 13:17:54.614946 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:54.615020 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 13:17:54.615347 17152 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 13:17:54.616076 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 16983
I20260619 13:17:54.616619 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37467
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34051
--webserver_port=36961
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:17:54.925793 17155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:17:54.926097 17155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:17:54.926232 17155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:17:54.933251 17155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:17:54.933424 17155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:17:54.933557 17155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:17:54.942899 17155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36961
--tserver_master_addrs=127.0.97.62:34051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:17:54.944039 17155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:17:54.945294 17155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:17:54.959299 17161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:54.959404 17160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:54.959383 17163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:17:54.960196 17155 server_base.cc:1034] running on GCE node
I20260619 13:17:54.960716 17155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:17:54.961561 17155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:17:54.962757 17155 hybrid_clock.cc:648] HybridClock initialized: now 1781875074962708 us; error 28 us; skew 500 ppm
I20260619 13:17:54.963032 17155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:17:54 dist-test-slave-mkt3 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:17:54.965929 17155 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:17:54.967263 17155 webserver.cc:460] Webserver started at http://127.0.97.2:36961/ using document root <none> and password file <none>
I20260619 13:17:54.967878 17155 fs_manager.cc:359] Metadata directory not provided
I20260619 13:17:54.967974 17155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:17:54.972997 17155 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 13:17:54.975554 17170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:54.976833 17155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:17:54.976966 17155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "187b99c27439439b926caaa279a244ea"
format_stamp: "Formatted at 2026-06-19 13:17:09 on dist-test-slave-mkt3"
server_key: "dd6d0c1dbbeacc123f804406997c1ddd"
server_key_iv: "0df1b8017ca7ed6d49531e6aadcc9e88"
server_key_version: "encryptionkey@0"
I20260619 13:17:54.977900 17155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:17:55.004048 17155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:17:55.008105 17155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:17:55.008327 17155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:17:55.009259 17155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:17:55.010589 17155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:17:55.010672 17155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:55.010759 17155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:17:55.010813 17155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:55.039894 17155 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37467
I20260619 13:17:55.039971 17282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37467 every 8 connection(s)
I20260619 13:17:55.041307 17155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:17:55.046000 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:55.044142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:44151)
0619 13:17:55.044859 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:55.045073 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":272,"threads_started":1}
W20260619 13:17:55.046690 17283 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34051 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34051: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
I20260619 13:17:55.046855 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 17155
I20260619 13:17:55.047482 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38541
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34051
--webserver_port=45479
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:17:55.047533 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:55.047154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:45779)
0619 13:17:55.047315 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:55.047397 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 13:17:55.048380 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:55.048009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:51847)
0619 13:17:55.048144 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:55.048242 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 13:17:55.048629 17283 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 13:17:55.353724 17286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:17:55.353991 17286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:17:55.354131 17286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:17:55.361238 17286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:17:55.361362 17286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:17:55.361543 17286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:17:55.371223 17286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45479
--tserver_master_addrs=127.0.97.62:34051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:17:55.372349 17286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:17:55.373500 17286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:17:55.385517 17291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:55.385393 17292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:17:55.385425 17294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:17:55.386379 17286 server_base.cc:1034] running on GCE node
I20260619 13:17:55.386910 17286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:17:55.387745 17286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:17:55.388955 17286 hybrid_clock.cc:648] HybridClock initialized: now 1781875075388909 us; error 41 us; skew 500 ppm
I20260619 13:17:55.389261 17286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:17:55 dist-test-slave-mkt3 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:17:55.392088 17286 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:17:55.393426 17286 webserver.cc:460] Webserver started at http://127.0.97.3:45479/ using document root <none> and password file <none>
I20260619 13:17:55.394101 17286 fs_manager.cc:359] Metadata directory not provided
I20260619 13:17:55.394217 17286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:17:55.399139 17286 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 13:17:55.401633 17301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:55.403098 17286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:17:55.403278 17286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "27f18f77dbd74c30a72c4d048de58669"
format_stamp: "Formatted at 2026-06-19 13:17:10 on dist-test-slave-mkt3"
server_key: "771c5a9accb251e73fcfc85a51ca907e"
server_key_iv: "5c44bf437847cb2e438d3f8dfb6dc2f0"
server_key_version: "encryptionkey@0"
I20260619 13:17:55.404188 17286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:17:55.427145 17286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:17:55.431036 17286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:17:55.431299 17286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:17:55.432231 17286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:17:55.433544 17286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:17:55.433651 17286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:55.433751 17286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:17:55.433820 17286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:17:55.462561 17286 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38541
I20260619 13:17:55.462630 17413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38541 every 8 connection(s)
I20260619 13:17:55.463909 17286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:17:55.468006 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 17286
I20260619 13:17:55.468659 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:55.466598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:41189)
0619 13:17:55.467290 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:55.467490 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":261,"threads_started":1}
W20260619 13:17:55.469305 17414 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34051 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34051: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
I20260619 13:17:55.470093 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:55.469775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:56199)
0619 13:17:55.469933 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:55.469995 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 13:17:55.470788 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:55.470509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:46429)
0619 13:17:55.470609 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:55.470675 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 13:17:55.470973 17414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 13:17:55.616623 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:55.615852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:36485)
0619 13:17:55.616304 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:55.616397 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":200,"threads_started":1}
I20260619 13:17:56.049829 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:56.049099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:42069)
0619 13:17:56.049522 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:56.049603 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":213,"threads_started":1}
I20260619 13:17:56.472115 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:56.471423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:45457)
0619 13:17:56.471827 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:56.471907 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":199,"threads_started":1}
I20260619 13:17:56.618023 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:56.617402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:42149)
0619 13:17:56.617781 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:56.617879 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":197,"threads_started":1}
I20260619 13:17:57.051283 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:57.050600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:39255)
0619 13:17:57.051044 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:57.051118 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":229,"threads_started":1}
I20260619 13:17:57.473567 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:57.472887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:40287)
0619 13:17:57.473322 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:57.473399 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":214,"threads_started":1}
I20260619 13:17:57.619510 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:57.618802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:43199)
0619 13:17:57.619262 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:57.619345 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":223,"threads_started":1}
I20260619 13:17:58.052793 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:58.052123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:43997)
0619 13:17:58.052535 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:58.052614 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":210,"threads_started":1}
I20260619 13:17:58.475000 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:58.474308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:48077)
0619 13:17:58.474751 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:58.474833 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":229,"threads_started":1}
I20260619 13:17:58.620983 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:58.620299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:45913)
0619 13:17:58.620743 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:58.620823 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":219,"threads_started":1}
I20260619 13:17:59.054358 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:59.053629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:54169)
0619 13:17:59.054095 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:59.054174 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260619 13:17:59.476514 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:59.475793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:55321)
0619 13:17:59.476246 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:59.476325 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":202,"threads_started":1}
I20260619 13:17:59.622574 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:17:59.621896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:33563)
0619 13:17:59.622326 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 13:17:59.622409 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":205,"threads_started":1}
I20260619 13:18:00.056005 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:00.055256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:47789)
0619 13:18:00.055752 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:00.055837 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":241,"threads_started":1}
I20260619 13:18:00.478161 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:00.477399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:50671)
0619 13:18:00.477915 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:00.477994 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":246,"threads_started":1}
I20260619 13:18:00.624099 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:00.623425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:46631)
0619 13:18:00.623867 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:00.623947 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":229,"threads_started":1}
I20260619 13:18:01.057622 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:01.056920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:39881)
0619 13:18:01.057358 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:01.057464 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":225,"threads_started":1}
I20260619 13:18:01.479781 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:01.479037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:48919)
0619 13:18:01.479482 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:01.479589 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":249,"threads_started":1}
I20260619 13:18:01.625798 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:01.624968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:36737)
0619 13:18:01.625468 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:01.625547 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":220,"threads_started":1}
I20260619 13:18:02.059331 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:02.058496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:47133)
0619 13:18:02.058994 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:02.059071 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":234,"threads_started":1}
I20260619 13:18:02.481333 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:02.480560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:34007)
0619 13:18:02.481020 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:02.481112 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":191,"threads_started":1}
I20260619 13:18:02.627414 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:02.626700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:44077)
0619 13:18:02.627167 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:02.627251 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":237,"threads_started":1}
I20260619 13:18:03.060834 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:03.060183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:37477)
0619 13:18:03.060595 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:03.060681 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":214,"threads_started":1}
I20260619 13:18:03.482828 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:03.482189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:58763)
0619 13:18:03.482591 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:03.482663 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":199,"threads_started":1}
I20260619 13:18:03.628891 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:03.628181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:36119)
0619 13:18:03.628625 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:03.628726 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":217,"threads_started":1}
I20260619 13:18:04.062579 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:04.061810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:33811)
0619 13:18:04.062317 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:04.062408 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":236,"threads_started":1}
I20260619 13:18:04.484434 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:04.483697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:41883)
0619 13:18:04.484178 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:04.484267 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":216,"threads_started":1}
I20260619 13:18:04.630537 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:04.629802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:32905)
0619 13:18:04.630288 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:04.630375 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":232,"threads_started":1}
I20260619 13:18:05.063997 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:05.063364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:60897)
0619 13:18:05.063763 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:05.063842 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":209,"threads_started":1}
I20260619 13:18:05.486128 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:05.485343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:54135)
0619 13:18:05.485883 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:05.485964 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":276,"threads_started":1}
I20260619 13:18:05.631985 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:05.631293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:53229)
0619 13:18:05.631731 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:05.631810 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":231,"threads_started":1}
I20260619 13:18:06.065572 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:06.064857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:45931)
0619 13:18:06.065310 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:06.065400 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":236,"threads_started":1}
I20260619 13:18:06.487684 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:06.486942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:48717)
0619 13:18:06.487395 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:06.487504 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":210,"threads_started":1}
I20260619 13:18:06.633392 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:06.632755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:35157)
0619 13:18:06.633140 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:06.633214 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":197,"threads_started":1}
I20260619 13:18:07.067130 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:07.066478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:51487)
0619 13:18:07.066892 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:07.066977 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":199,"threads_started":1}
I20260619 13:18:07.489137 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:07.488471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:50191)
0619 13:18:07.488907 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:07.488992 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":222,"threads_started":1}
I20260619 13:18:07.635119 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:07.634274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:60055)
0619 13:18:07.634787 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:07.634883 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":226,"threads_started":1}
I20260619 13:18:08.068744 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:08.067998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:60713)
0619 13:18:08.068473 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:08.068559 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":235,"threads_started":1}
I20260619 13:18:08.490767 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:08.490052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:60933)
0619 13:18:08.490488 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:08.490566 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":205,"threads_started":1}
I20260619 13:18:08.636655 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:08.635938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:49327)
0619 13:18:08.636395 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:08.636475 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260619 13:18:09.070392 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:09.069570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:37397)
0619 13:18:09.070126 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:09.070211 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":349,"threads_started":1}
I20260619 13:18:09.492148 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:09.491513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:49379)
0619 13:18:09.491899 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:09.491974 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":190,"threads_started":1}
I20260619 13:18:09.638392 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:09.637679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.1:42249)
0619 13:18:09.638129 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:09.638208 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":235,"threads_started":1}
I20260619 13:18:10.072115 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:10.071361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.2:39909)
0619 13:18:10.071840 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:10.071930 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":246,"threads_started":1}
I20260619 13:18:10.477833 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 16983
I20260619 13:18:10.493922 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:10.493161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34051 (local address 127.0.97.3:53043)
0619 13:18:10.493612 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:10.493697 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":247,"threads_started":1}
I20260619 13:18:10.496793 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:10.478888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.0.97.1:44927)
0619 13:18:10.479387 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:10.479399 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:10.479436 (+ 37us) server_negotiation.cc:407] Connection header received
0619 13:18:10.481788 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:10.481802 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:10.482395 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:10.482780 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:10.485386 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:10.486711 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:10.487921 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:10.488582 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:10.488709 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:10.491131 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:10.491139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:10.491154 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:10.491194 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:10.493864 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:10.494551 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:10.494558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:10.494564 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:10.494724 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:10.495016 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:10.495018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:10.495020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:10.495513 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:10.495733 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:10.496162 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 13:18:10.496533 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":236,"threads_started":1}
I20260619 13:18:10.498131 17147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:10.581002 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 16983
I20260619 13:18:10.594408 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 17155
I20260619 13:18:10.610879 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:10.595221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.0.97.2:37467)
0619 13:18:10.595703 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:10.595718 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:10.595756 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:18:10.597189 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:10.597199 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:10.597746 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:10.598109 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:10.599391 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:10.600645 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:10.602023 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:10.602673 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:10.602799 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:10.605059 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:10.605066 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:10.605080 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:10.605116 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:10.607986 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:10.608672 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:10.608679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:10.608685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:10.608842 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:10.609199 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:10.609201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:10.609203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:10.609667 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:10.609917 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:10.610295 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 13:18:10.610608 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":211,"threads_started":1}
I20260619 13:18:10.612215 17278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:10.692822 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 17155
I20260619 13:18:10.706806 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 17286
I20260619 13:18:10.724478 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:10.707754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.0.97.3:38541)
0619 13:18:10.708337 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:10.708351 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:10.708390 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:18:10.710561 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:10.710573 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:10.711128 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:10.711505 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:10.712871 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:10.714113 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:10.715313 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:10.715972 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:10.716154 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:10.718684 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:10.718692 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:10.718707 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:10.718742 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:10.721374 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:10.722041 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:10.722049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:10.722058 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:10.722221 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:10.722576 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:10.722581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:10.722585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:10.723079 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:10.723355 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:10.723800 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 13:18:10.724192 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":278,"threads_started":1}
I20260619 13:18:10.725744 17409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:10.805723 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 17286
2026-06-19T13:18:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63564 ms)
[----------] 30 tests from SecurityITest (283469 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:10 dist-test-slave-mkt3 krb5kdc[17471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:10 dist-test-slave-mkt3 krb5kdc[17471](info): set up 2 sockets
Jun 19 13:18:10 dist-test-slave-mkt3 krb5kdc[17471](info): commencing operation
krb5kdc: starting...
W20260619 13:18:11.817560 388 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:11 dist-test-slave-mkt3 krb5kdc[17471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:18:12.038908 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43791
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:12.352595 17484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:12.352871 17484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:12.352957 17484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:12.359869 17484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:12.359962 17484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:12.360013 17484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:12.360069 17484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:12.360111 17484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:12.369385 17484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:12.370535 17484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:12.371734 17484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:12.382427 17490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:12.382488 17489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:12.382457 17492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:12.383394 17484 server_base.cc:1034] running on GCE node
I20260619 13:18:12.384052 17484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:12.385300 17484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:12.386500 17484 hybrid_clock.cc:648] HybridClock initialized: now 1781875092386444 us; error 40 us; skew 500 ppm
I20260619 13:18:12.386861 17484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:12 dist-test-slave-mkt3 krb5kdc[17471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:12.389768 17484 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:18:12.391111 17484 webserver.cc:460] Webserver started at http://127.0.97.62:39573/ using document root <none> and password file <none>
I20260619 13:18:12.391741 17484 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:12.391846 17484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:12.392127 17484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:12.394937 17484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "8fdab99704374d72b1d69a2fd5bcc340"
format_stamp: "Formatted at 2026-06-19 13:18:12 on dist-test-slave-mkt3"
server_key: "16eafa47f256b6010c9bfc507e75db7d"
server_key_iv: "bb4338dfacc4c39d828686d46ef777fd"
server_key_version: "encryptionkey@0"
I20260619 13:18:12.395630 17484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "8fdab99704374d72b1d69a2fd5bcc340"
format_stamp: "Formatted at 2026-06-19 13:18:12 on dist-test-slave-mkt3"
server_key: "16eafa47f256b6010c9bfc507e75db7d"
server_key_iv: "bb4338dfacc4c39d828686d46ef777fd"
server_key_version: "encryptionkey@0"
I20260619 13:18:12.400435 17484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:18:12.403666 17499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:12.404908 17484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:18:12.405061 17484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "8fdab99704374d72b1d69a2fd5bcc340"
format_stamp: "Formatted at 2026-06-19 13:18:12 on dist-test-slave-mkt3"
server_key: "16eafa47f256b6010c9bfc507e75db7d"
server_key_iv: "bb4338dfacc4c39d828686d46ef777fd"
server_key_version: "encryptionkey@0"
I20260619 13:18:12.405228 17484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:12.427186 17484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:12.431134 17484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:12.431382 17484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:12.447566 17484 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43791
I20260619 13:18:12.447597 17550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43791 every 8 connection(s)
I20260619 13:18:12.448873 17484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:12.450414 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 17484
I20260619 13:18:12.450614 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:12.451378 388 external_mini_cluster.cc:1428] Setting key 3cc0d06dd87c9c2b26b1d67a545ff157
I20260619 13:18:12.453303 17551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 13:18:12 dist-test-slave-mkt3 krb5kdc[17471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:18:12.465721 17551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340: Bootstrap starting.
I20260619 13:18:12.468652 17551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:12.469583 17551 log.cc:824] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:12.472496 17551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340: No bootstrap required, opened a new log
I20260619 13:18:12.473001 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:12.453286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39362 (local address 127.0.97.62:43791)
0619 13:18:12.454027 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:12.454038 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:12.454081 (+ 43us) server_negotiation.cc:407] Connection header received
0619 13:18:12.455669 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:12.455690 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:12.456074 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:12.456395 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:12.458060 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:12.459327 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:12.461113 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:12.461716 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:12.461897 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:12.466113 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:12.466120 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:12.466133 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:12.466166 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:12.468948 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:12.469600 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:12.469606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:12.469612 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:12.469803 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:12.470212 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:12.470216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:12.470218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:12.470754 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:12.471025 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:12.471355 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 13:18:12.471752 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":308,"threads_started":1}
I20260619 13:18:12.479570 17551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdab99704374d72b1d69a2fd5bcc340" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43791 } }
I20260619 13:18:12.479866 17551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:12.479952 17551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fdab99704374d72b1d69a2fd5bcc340, State: Initialized, Role: FOLLOWER
I20260619 13:18:12.480497 17551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdab99704374d72b1d69a2fd5bcc340" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43791 } }
I20260619 13:18:12.480710 17551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:12.480834 17551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:12.480983 17551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:12.482996 17551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdab99704374d72b1d69a2fd5bcc340" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43791 } }
I20260619 13:18:12.483431 17551 leader_election.cc:302] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fdab99704374d72b1d69a2fd5bcc340; no voters:
I20260619 13:18:12.483923 17551 leader_election.cc:288] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:12.484030 17556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:12.484616 17556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [term 1 LEADER]: Becoming Leader. State: Replica: 8fdab99704374d72b1d69a2fd5bcc340, State: Running, Role: LEADER
I20260619 13:18:12.485140 17556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdab99704374d72b1d69a2fd5bcc340" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43791 } }
I20260619 13:18:12.485684 17551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:12.488051 17557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fdab99704374d72b1d69a2fd5bcc340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdab99704374d72b1d69a2fd5bcc340" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43791 } } }
I20260619 13:18:12.488433 17557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:12.488457 17558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fdab99704374d72b1d69a2fd5bcc340. Latest consensus state: current_term: 1 leader_uuid: "8fdab99704374d72b1d69a2fd5bcc340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdab99704374d72b1d69a2fd5bcc340" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43791 } } }
I20260619 13:18:12.488724 17558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:12.489971 17565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:12.494817 17565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:12.503162 17565 catalog_manager.cc:1347] Generated new cluster ID: c8e60ac865864805b3da229ea9618014
I20260619 13:18:12.503265 17565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:12.525945 17565 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:12.527734 17565 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:12.535475 17565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340: Generated new TSK 0
I20260619 13:18:12.536391 17565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:18:12.584549 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43791
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:12.895635 17577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:12.895908 17577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:12.896046 17577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:12.903270 17577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:12.903367 17577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:12.903512 17577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:18:12.913179 17577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:12.914451 17577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:12.915722 17577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:12.928047 17582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:12.928078 17583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:12.928010 17585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:12.928900 17577 server_base.cc:1034] running on GCE node
I20260619 13:18:12.929486 17577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:12.930475 17577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:12.931715 17577 hybrid_clock.cc:648] HybridClock initialized: now 1781875092931650 us; error 38 us; skew 500 ppm
I20260619 13:18:12.932058 17577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:12 dist-test-slave-mkt3 krb5kdc[17471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:12.935266 17577 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:18:12.936647 17577 webserver.cc:460] Webserver started at http://127.0.97.1:38247/ using document root <none> and password file <none>
I20260619 13:18:12.937343 17577 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:12.937448 17577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:12.937740 17577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:12.940578 17577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "dd1ba464cba745c0be8a12a38ca2136f"
format_stamp: "Formatted at 2026-06-19 13:18:12 on dist-test-slave-mkt3"
server_key: "872a24e91bd7f31d09a68f714474b4a7"
server_key_iv: "890d322711345b30f482eb469df7741b"
server_key_version: "encryptionkey@0"
I20260619 13:18:12.941327 17577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dd1ba464cba745c0be8a12a38ca2136f"
format_stamp: "Formatted at 2026-06-19 13:18:12 on dist-test-slave-mkt3"
server_key: "872a24e91bd7f31d09a68f714474b4a7"
server_key_iv: "890d322711345b30f482eb469df7741b"
server_key_version: "encryptionkey@0"
I20260619 13:18:12.946120 17577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260619 13:18:12.949977 17592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:12.951550 17577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:18:12.951717 17577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "dd1ba464cba745c0be8a12a38ca2136f"
format_stamp: "Formatted at 2026-06-19 13:18:12 on dist-test-slave-mkt3"
server_key: "872a24e91bd7f31d09a68f714474b4a7"
server_key_iv: "890d322711345b30f482eb469df7741b"
server_key_version: "encryptionkey@0"
I20260619 13:18:12.951921 17577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:12.970638 17577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:12.974337 17577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:12.974553 17577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:12.975409 17577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:12.976612 17577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:12.976678 17577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:12.976773 17577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:12.976815 17577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:13.005985 17577 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43531
I20260619 13:18:13.006039 17704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43531 every 8 connection(s)
I20260619 13:18:13.007277 17577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:18:13.014876 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 17577
I20260619 13:18:13.015084 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:18:13.015837 388 external_mini_cluster.cc:1428] Setting key ad000ec331fdd937238ca55b6e5e9e8d
Jun 19 13:18:13 dist-test-slave-mkt3 krb5kdc[17471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:18:13.027858 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:13.009780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58297 (local address 127.0.97.62:43791)
0619 13:18:13.010233 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:13.010238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:13.011382 (+ 1144us) server_negotiation.cc:407] Connection header received
0619 13:18:13.013313 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:13.013317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:13.013392 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:13.013487 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:13.014990 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:13.016059 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:13.017817 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:13.018416 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:13.018541 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:13.021863 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:13.021870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:13.021874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:13.021914 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:13.024785 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:13.025514 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:13.025518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:13.025520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:13.025645 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:13.026115 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:13.026119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:13.026121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:13.026427 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:13.026568 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:13.027276 (+ 708us) server_negotiation.cc:299] Negotiation successful
0619 13:18:13.027491 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":223,"threads_started":1}
I20260619 13:18:13.028805 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:13.010396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43791 (local address 127.0.97.1:58297)
0619 13:18:13.011169 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:13.011252 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:13.013040 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:13.013686 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:13.013695 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:13.014277 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:13.014819 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:13.014837 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:13.016248 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:13.016252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:13.017669 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:13.017685 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:13.018647 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:13.018653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:13.018845 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:13.019510 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:18:13.019535 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:13.021622 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:13.024919 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:13.024926 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:13.024939 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:13.025386 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:13.025741 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:13.025745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:13.025748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:13.026002 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:13.026558 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:13.026565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:13.026913 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 13:18:13.027157 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 13:18:13.027670 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":253,"threads_started":1}
I20260619 13:18:13.030287 17705 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43791
I20260619 13:18:13.030551 17705 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:13.031221 17705 heartbeater.cc:511] Master 127.0.97.62:43791 requested a full tablet report, sending...
I20260619 13:18:13.033329 17516 ts_manager.cc:194] Registered new tserver with Master: dd1ba464cba745c0be8a12a38ca2136f (127.0.97.1:43531)
I20260619 13:18:13.035322 17516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58297
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:18:13.063055 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43791
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:13.370434 17711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:13.370723 17711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:13.370868 17711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:13.378049 17711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:13.378165 17711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:13.378314 17711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:18:13.388619 17711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:13.389982 17711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:13.391270 17711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:13.403663 17719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:13.403640 17717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:13.404069 17716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:13.404450 17711 server_base.cc:1034] running on GCE node
I20260619 13:18:13.405014 17711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:13.405948 17711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:13.407181 17711 hybrid_clock.cc:648] HybridClock initialized: now 1781875093407120 us; error 49 us; skew 500 ppm
I20260619 13:18:13.407496 17711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:13 dist-test-slave-mkt3 krb5kdc[17471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:13.410624 17711 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:18:13.412128 17711 webserver.cc:460] Webserver started at http://127.0.97.2:42857/ using document root <none> and password file <none>
I20260619 13:18:13.412966 17711 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:13.413087 17711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:13.413383 17711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:13.416327 17711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "66b8b032e77f4dabbc10226a77ce185a"
format_stamp: "Formatted at 2026-06-19 13:18:13 on dist-test-slave-mkt3"
server_key: "256ec45d9798a02314dc75215391d5e0"
server_key_iv: "341b7aa954b1a8d96f01d76be63875e7"
server_key_version: "encryptionkey@0"
I20260619 13:18:13.417088 17711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "66b8b032e77f4dabbc10226a77ce185a"
format_stamp: "Formatted at 2026-06-19 13:18:13 on dist-test-slave-mkt3"
server_key: "256ec45d9798a02314dc75215391d5e0"
server_key_iv: "341b7aa954b1a8d96f01d76be63875e7"
server_key_version: "encryptionkey@0"
I20260619 13:18:13.422308 17711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:18:13.425956 17726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:13.427359 17711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:18:13.427520 17711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "66b8b032e77f4dabbc10226a77ce185a"
format_stamp: "Formatted at 2026-06-19 13:18:13 on dist-test-slave-mkt3"
server_key: "256ec45d9798a02314dc75215391d5e0"
server_key_iv: "341b7aa954b1a8d96f01d76be63875e7"
server_key_version: "encryptionkey@0"
I20260619 13:18:13.427732 17711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:13.443317 17711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:13.447438 17711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:13.447751 17711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:13.448871 17711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:13.450485 17711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:13.450572 17711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:13.450682 17711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:13.450760 17711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:13.480764 17711 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42883
I20260619 13:18:13.480806 17838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42883 every 8 connection(s)
I20260619 13:18:13.481997 17711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:18:13.482496 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 17711
I20260619 13:18:13.482668 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:18:13.483201 388 external_mini_cluster.cc:1428] Setting key 0f44ee77bdb28a093ef65f0b79bbffca
Jun 19 13:18:13 dist-test-slave-mkt3 krb5kdc[17471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:18:13.501001 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:13.484585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39013 (local address 127.0.97.62:43791)
0619 13:18:13.484767 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:13.484773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:13.486041 (+ 1268us) server_negotiation.cc:407] Connection header received
0619 13:18:13.487889 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:13.487894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:13.487987 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:13.488072 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:13.489614 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:13.490407 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:13.491885 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:13.492278 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:13.492384 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:13.495745 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:13.495750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:13.495753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:13.495785 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:13.498176 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:13.498832 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:13.498837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:13.498839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:13.498964 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:13.499362 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:13.499367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:13.499371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:13.499629 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:13.499770 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:13.500429 (+ 659us) server_negotiation.cc:299] Negotiation successful
0619 13:18:13.500658 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:18:13.501983 17841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:13.485137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43791 (local address 127.0.97.2:39013)
0619 13:18:13.485895 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:13.485946 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:13.487642 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:13.488315 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:13.488324 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:13.488908 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:13.489457 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:13.489475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:13.490555 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:13.490559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:13.491749 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:13.491762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:13.492482 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:13.492489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:13.492701 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:13.493355 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:18:13.493376 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:13.495542 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:13.498303 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:13.498312 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:13.498329 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:13.498719 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:13.499065 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:13.499070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:13.499074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:13.499260 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:13.499749 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:13.499757 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:13.500078 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 13:18:13.500329 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 13:18:13.500845 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":305,"threads_started":1}
I20260619 13:18:13.503393 17839 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43791
I20260619 13:18:13.503666 17839 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:18:13.504376 17839 heartbeater.cc:511] Master 127.0.97.62:43791 requested a full tablet report, sending...
I20260619 13:18:13.505893 17516 ts_manager.cc:194] Registered new tserver with Master: 66b8b032e77f4dabbc10226a77ce185a (127.0.97.2:42883)
I20260619 13:18:13.506798 17516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:18:13.531131 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43791
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:13.839711 17844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:13.839968 17844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:13.840075 17844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:13.847126 17844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:13.847227 17844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:13.847345 17844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:18:13.856619 17844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:13.857765 17844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:13.858985 17844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:13.871073 17849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:13.870996 17852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:13.871043 17850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:13.871724 17844 server_base.cc:1034] running on GCE node
I20260619 13:18:13.872264 17844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:13.873126 17844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:13.874315 17844 hybrid_clock.cc:648] HybridClock initialized: now 1781875093874266 us; error 33 us; skew 500 ppm
I20260619 13:18:13.874594 17844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:13 dist-test-slave-mkt3 krb5kdc[17471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:13.877465 17844 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:18:13.878834 17844 webserver.cc:460] Webserver started at http://127.0.97.3:44267/ using document root <none> and password file <none>
I20260619 13:18:13.879473 17844 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:13.879583 17844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:13.879879 17844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:13.882642 17844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "9801dd68b189432682801bd66638ce97"
format_stamp: "Formatted at 2026-06-19 13:18:13 on dist-test-slave-mkt3"
server_key: "f64ce5a800b9083f97c74399fd4e092c"
server_key_iv: "0fb95f7ebbaf00d3649c02e6a32879f1"
server_key_version: "encryptionkey@0"
I20260619 13:18:13.883366 17844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9801dd68b189432682801bd66638ce97"
format_stamp: "Formatted at 2026-06-19 13:18:13 on dist-test-slave-mkt3"
server_key: "f64ce5a800b9083f97c74399fd4e092c"
server_key_iv: "0fb95f7ebbaf00d3649c02e6a32879f1"
server_key_version: "encryptionkey@0"
I20260619 13:18:13.888240 17844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:18:13.891585 17859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:13.892827 17844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:18:13.892962 17844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "9801dd68b189432682801bd66638ce97"
format_stamp: "Formatted at 2026-06-19 13:18:13 on dist-test-slave-mkt3"
server_key: "f64ce5a800b9083f97c74399fd4e092c"
server_key_iv: "0fb95f7ebbaf00d3649c02e6a32879f1"
server_key_version: "encryptionkey@0"
I20260619 13:18:13.893128 17844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:13.907306 17844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:13.911342 17844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:13.911593 17844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:13.912482 17844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:13.913762 17844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:13.913857 17844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:13.913976 17844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:13.914050 17844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:13.942767 17844 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46239
I20260619 13:18:13.942802 17971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46239 every 8 connection(s)
I20260619 13:18:13.943967 17844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:18:13.951623 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 17844
I20260619 13:18:13.951785 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:18:13.952311 388 external_mini_cluster.cc:1428] Setting key dc66cf822a932215bded69b3d7642306
Jun 19 13:18:13 dist-test-slave-mkt3 krb5kdc[17471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:18:13.962522 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:13.946344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50793 (local address 127.0.97.62:43791)
0619 13:18:13.946514 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:13.946518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:13.947702 (+ 1184us) server_negotiation.cc:407] Connection header received
0619 13:18:13.949451 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:13.949457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:13.949534 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:13.949621 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:13.951144 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:13.952169 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:13.953681 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:13.954309 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:13.954401 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:13.957765 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:13.957769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:13.957773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:13.957806 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:13.960119 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:13.960742 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:13.960746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:13.960748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:13.960853 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:13.961178 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:13.961181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:13.961183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:13.961422 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:13.961527 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:13.962123 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 13:18:13.962282 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:18:13.963387 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:13.946838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43791 (local address 127.0.97.3:50793)
0619 13:18:13.947559 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:13.947596 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:13.949188 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:13.949883 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:13.949895 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:13.950437 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:13.950977 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:13.950995 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:13.952317 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:13.952321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:13.953545 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:13.953559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:13.954519 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:13.954526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:13.954823 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:13.955527 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:18:13.955549 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:13.957574 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:13.960243 (+ 2669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:13.960249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:13.960260 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:13.960653 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:13.960924 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:13.960927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:13.960929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:13.961103 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:13.961506 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:13.961511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:13.961791 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 13:18:13.962037 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 13:18:13.962379 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":260,"threads_started":1}
I20260619 13:18:13.964583 17972 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43791
I20260619 13:18:13.964845 17972 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:13.965494 17972 heartbeater.cc:511] Master 127.0.97.62:43791 requested a full tablet report, sending...
I20260619 13:18:13.966858 17516 ts_manager.cc:194] Registered new tserver with Master: 9801dd68b189432682801bd66638ce97 (127.0.97.3:46239)
I20260619 13:18:13.967361 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:18:13.967677 17516 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50793
I20260619 13:18:13.982846 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:13.971439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.0.97.62:43791)
0619 13:18:13.971578 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:13.971582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:13.972077 (+ 495us) server_negotiation.cc:407] Connection header received
0619 13:18:13.972215 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:13.972219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:13.972269 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:13.972344 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:13.973638 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:13.974355 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:13.975700 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:13.976076 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:13.976161 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:13.978400 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:13.978404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:13.978407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:13.978432 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:13.980736 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:13.981313 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:13.981316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:13.981318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:13.981417 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:13.981682 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:13.981685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:13.981687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:13.982003 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:13.982171 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:13.982468 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 13:18:13.982613 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:18:13.986503 17516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:18:13.993331 17516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:18:14.023325 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:14.005693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.0.97.2:42883)
0619 13:18:14.016584 (+ 10891us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:14.016592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:14.016621 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:18:14.016714 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:14.016722 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:14.016956 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:14.017147 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:14.017665 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.018893 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.021333 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.022278 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.022400 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:14.022591 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:14.022858 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 13:18:14.023022 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10716,"thread_start_us":10074,"threads_started":1}
I20260619 13:18:14.023432 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:14.005487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43531 (local address 127.0.0.1:60228)
0619 13:18:14.014289 (+ 8802us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:14.014312 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:14.014423 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:14.015967 (+ 1544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:14.015975 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:14.016006 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:14.016453 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:14.016466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.018338 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.018343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:14.020750 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:14.020762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.022554 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.022561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:14.022733 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:14.022764 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:18:14.022837 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 13:18:14.023022 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8675,"mutex_wait_us":221,"thread_start_us":362,"threads_started":1}
I20260619 13:18:14.023818 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:14.007146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46239 (local address 127.0.0.1:59476)
0619 13:18:14.015385 (+ 8239us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:14.015403 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:14.015541 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:14.016098 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:14.016102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:14.016131 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:14.016671 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:14.016685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.018458 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.018462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:14.020928 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:14.020942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.022838 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.022844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:14.022992 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:14.023020 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 13:18:14.023109 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 13:18:14.023229 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8115,"mutex_wait_us":152,"thread_start_us":6884,"threads_started":1}
I20260619 13:18:14.023998 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:14.006010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.0.97.1:43531)
0619 13:18:14.015361 (+ 9351us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:14.015369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:14.015394 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:18:14.015476 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:14.015483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:14.015732 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:14.016673 (+ 941us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:14.016914 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.018221 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.020892 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.022307 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.022952 (+ 645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:14.023154 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:14.023221 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 13:18:14.023364 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9194,"thread_start_us":9066,"threads_started":1}
I20260619 13:18:14.024503 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:14.007675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.0.97.3:46239)
0619 13:18:14.014995 (+ 7320us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:14.015004 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:14.015505 (+ 501us) server_negotiation.cc:407] Connection header received
0619 13:18:14.015683 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:14.015690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:14.015903 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:14.016082 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:14.016879 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.018331 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.021087 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.022372 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.022492 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:14.022703 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:14.024103 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0619 13:18:14.024248 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7145,"thread_start_us":6985,"threads_started":1}
I20260619 13:18:14.025141 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:14.005297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42883 (local address 127.0.0.1:52920)
0619 13:18:14.012237 (+ 6940us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:14.012286 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:14.012465 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:14.017153 (+ 4688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:14.017157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:14.017181 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:14.017511 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:14.017523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.018992 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.018997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:14.021197 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:14.021214 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.022464 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.022473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:14.022656 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:14.022697 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 13:18:14.022810 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 13:18:14.023912 (+ 1102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6652,"mutex_wait_us":37,"thread_start_us":9829,"threads_started":1}
I20260619 13:18:14.027420 17640 tablet_service.cc:1467] Processing CreateTablet for tablet 01019109e8f44a358bd22df26fc5018d (DEFAULT_TABLE table=test-table [id=76d9ed1180ef4ac2b30dd0f423b62f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:14.027774 17907 tablet_service.cc:1467] Processing CreateTablet for tablet 01019109e8f44a358bd22df26fc5018d (DEFAULT_TABLE table=test-table [id=76d9ed1180ef4ac2b30dd0f423b62f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:14.028625 17640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01019109e8f44a358bd22df26fc5018d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:14.028314 17774 tablet_service.cc:1467] Processing CreateTablet for tablet 01019109e8f44a358bd22df26fc5018d (DEFAULT_TABLE table=test-table [id=76d9ed1180ef4ac2b30dd0f423b62f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:14.029170 17907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01019109e8f44a358bd22df26fc5018d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:14.029573 17774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01019109e8f44a358bd22df26fc5018d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:14.038200 17705 heartbeater.cc:503] Master 127.0.97.62:43791 was elected leader, sending a full tablet report...
I20260619 13:18:14.040836 17990 tablet_bootstrap.cc:492] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f: Bootstrap starting.
I20260619 13:18:14.043164 17991 tablet_bootstrap.cc:492] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a: Bootstrap starting.
I20260619 13:18:14.045397 17990 tablet_bootstrap.cc:654] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:14.046255 17992 tablet_bootstrap.cc:492] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97: Bootstrap starting.
I20260619 13:18:14.046504 17991 tablet_bootstrap.cc:654] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:14.046670 17990 log.cc:824] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:14.047637 17991 log.cc:824] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:14.049569 17992 tablet_bootstrap.cc:654] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:14.049600 17990 tablet_bootstrap.cc:492] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f: No bootstrap required, opened a new log
I20260619 13:18:14.049883 17990 ts_tablet_manager.cc:1397] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 13:18:14.050665 17992 log.cc:824] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:14.050769 17991 tablet_bootstrap.cc:492] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a: No bootstrap required, opened a new log
I20260619 13:18:14.051016 17991 ts_tablet_manager.cc:1397] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260619 13:18:14.055909 17992 tablet_bootstrap.cc:492] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97: No bootstrap required, opened a new log
I20260619 13:18:14.056146 17992 ts_tablet_manager.cc:1397] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260619 13:18:14.060576 17990 raft_consensus.cc:348] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b8b032e77f4dabbc10226a77ce185a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42883 } } peers { permanent_uuid: "dd1ba464cba745c0be8a12a38ca2136f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43531 } } peers { permanent_uuid: "9801dd68b189432682801bd66638ce97" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46239 } }
I20260619 13:18:14.061017 17990 raft_consensus.cc:374] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:14.061124 17990 raft_consensus.cc:729] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1ba464cba745c0be8a12a38ca2136f, State: Initialized, Role: FOLLOWER
I20260619 13:18:14.060855 17991 raft_consensus.cc:348] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b8b032e77f4dabbc10226a77ce185a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42883 } } peers { permanent_uuid: "dd1ba464cba745c0be8a12a38ca2136f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43531 } } peers { permanent_uuid: "9801dd68b189432682801bd66638ce97" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46239 } }
I20260619 13:18:14.061262 17991 raft_consensus.cc:374] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:14.061352 17991 raft_consensus.cc:729] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66b8b032e77f4dabbc10226a77ce185a, State: Initialized, Role: FOLLOWER
I20260619 13:18:14.062093 17991 consensus_queue.cc:260] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b8b032e77f4dabbc10226a77ce185a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42883 } } peers { permanent_uuid: "dd1ba464cba745c0be8a12a38ca2136f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43531 } } peers { permanent_uuid: "9801dd68b189432682801bd66638ce97" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46239 } }
I20260619 13:18:14.062093 17990 consensus_queue.cc:260] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b8b032e77f4dabbc10226a77ce185a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42883 } } peers { permanent_uuid: "dd1ba464cba745c0be8a12a38ca2136f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43531 } } peers { permanent_uuid: "9801dd68b189432682801bd66638ce97" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46239 } }
I20260619 13:18:14.063180 17839 heartbeater.cc:503] Master 127.0.97.62:43791 was elected leader, sending a full tablet report...
I20260619 13:18:14.063856 17991 ts_tablet_manager.cc:1428] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260619 13:18:14.063697 17992 raft_consensus.cc:348] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b8b032e77f4dabbc10226a77ce185a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42883 } } peers { permanent_uuid: "dd1ba464cba745c0be8a12a38ca2136f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43531 } } peers { permanent_uuid: "9801dd68b189432682801bd66638ce97" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46239 } }
I20260619 13:18:14.064111 17992 raft_consensus.cc:374] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:14.064205 17992 raft_consensus.cc:729] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9801dd68b189432682801bd66638ce97, State: Initialized, Role: FOLLOWER
I20260619 13:18:14.064873 17992 consensus_queue.cc:260] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b8b032e77f4dabbc10226a77ce185a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42883 } } peers { permanent_uuid: "dd1ba464cba745c0be8a12a38ca2136f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43531 } } peers { permanent_uuid: "9801dd68b189432682801bd66638ce97" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46239 } }
I20260619 13:18:14.065138 17990 ts_tablet_manager.cc:1428] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260619 13:18:14.066190 17972 heartbeater.cc:503] Master 127.0.97.62:43791 was elected leader, sending a full tablet report...
I20260619 13:18:14.067764 17992 ts_tablet_manager.cc:1428] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260619 13:18:14.142216 17998 raft_consensus.cc:482] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:18:14.142509 17998 raft_consensus.cc:504] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b8b032e77f4dabbc10226a77ce185a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42883 } } peers { permanent_uuid: "dd1ba464cba745c0be8a12a38ca2136f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43531 } } peers { permanent_uuid: "9801dd68b189432682801bd66638ce97" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46239 } }
I20260619 13:18:14.144176 17998 leader_election.cc:288] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66b8b032e77f4dabbc10226a77ce185a (127.0.97.2:42883), dd1ba464cba745c0be8a12a38ca2136f (127.0.97.1:43531)
I20260619 13:18:14.149683 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:14.144560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42883 (local address 127.0.97.3:60815)
0619 13:18:14.144896 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:14.144916 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:14.145051 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:14.145381 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:14.145386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:14.145413 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:14.145616 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:14.145625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.146623 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.146627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:14.147881 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:14.147893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.148984 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.148989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:14.149152 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:14.149176 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 13:18:14.149263 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 13:18:14.149387 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"spinlock_wait_cycles":24448}
I20260619 13:18:14.149672 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:14.144577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60815 (local address 127.0.97.2:42883)
0619 13:18:14.144784 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:14.144790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:14.145015 (+ 225us) server_negotiation.cc:407] Connection header received
0619 13:18:14.145199 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:14.145203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:14.145268 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:14.145342 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:14.145759 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.146484 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.148039 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.148823 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.148946 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:14.149035 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:14.149306 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 13:18:14.149429 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:18:14.150131 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:14.144775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50595 (local address 127.0.97.1:43531)
0619 13:18:14.144965 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:14.144970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:14.145412 (+ 442us) server_negotiation.cc:407] Connection header received
0619 13:18:14.145581 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:14.145585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:14.145641 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:14.145711 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:14.146213 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.146917 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.148525 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.149276 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.149391 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:14.149486 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:14.149757 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 13:18:14.149911 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:18:14.150484 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:14.144618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43531 (local address 127.0.97.3:50595)
0619 13:18:14.145315 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:14.145329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:14.145449 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:14.145736 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:14.145740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:14.145762 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:14.146065 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:14.146074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.147064 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.147068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:14.148378 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:14.148389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.149466 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.149473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:14.149612 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:14.149634 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:18:14.149714 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 13:18:14.150164 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":2,"thread_start_us":399,"threads_started":1}
I20260619 13:18:14.150681 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01019109e8f44a358bd22df26fc5018d" candidate_uuid: "9801dd68b189432682801bd66638ce97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66b8b032e77f4dabbc10226a77ce185a" is_pre_election: true
I20260619 13:18:14.151230 17794 raft_consensus.cc:2454] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9801dd68b189432682801bd66638ce97 in term 0.
I20260619 13:18:14.151446 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01019109e8f44a358bd22df26fc5018d" candidate_uuid: "9801dd68b189432682801bd66638ce97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1ba464cba745c0be8a12a38ca2136f" is_pre_election: true
I20260619 13:18:14.151983 17660 raft_consensus.cc:2454] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9801dd68b189432682801bd66638ce97 in term 0.
I20260619 13:18:14.152004 17861 leader_election.cc:302] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66b8b032e77f4dabbc10226a77ce185a, 9801dd68b189432682801bd66638ce97; no voters:
I20260619 13:18:14.152362 17998 raft_consensus.cc:2781] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:18:14.152521 17998 raft_consensus.cc:482] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:18:14.152627 17998 raft_consensus.cc:3037] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:14.154882 17998 raft_consensus.cc:504] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b8b032e77f4dabbc10226a77ce185a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42883 } } peers { permanent_uuid: "dd1ba464cba745c0be8a12a38ca2136f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43531 } } peers { permanent_uuid: "9801dd68b189432682801bd66638ce97" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46239 } }
I20260619 13:18:14.155468 17998 leader_election.cc:288] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [CANDIDATE]: Term 1 election: Requested vote from peers 66b8b032e77f4dabbc10226a77ce185a (127.0.97.2:42883), dd1ba464cba745c0be8a12a38ca2136f (127.0.97.1:43531)
I20260619 13:18:14.155893 17794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01019109e8f44a358bd22df26fc5018d" candidate_uuid: "9801dd68b189432682801bd66638ce97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66b8b032e77f4dabbc10226a77ce185a"
I20260619 13:18:14.155948 17660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01019109e8f44a358bd22df26fc5018d" candidate_uuid: "9801dd68b189432682801bd66638ce97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1ba464cba745c0be8a12a38ca2136f"
I20260619 13:18:14.156111 17794 raft_consensus.cc:3037] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:14.156126 17660 raft_consensus.cc:3037] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:14.159240 17794 raft_consensus.cc:2454] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9801dd68b189432682801bd66638ce97 in term 1.
I20260619 13:18:14.159246 17660 raft_consensus.cc:2454] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9801dd68b189432682801bd66638ce97 in term 1.
I20260619 13:18:14.159659 17860 leader_election.cc:302] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9801dd68b189432682801bd66638ce97, dd1ba464cba745c0be8a12a38ca2136f; no voters:
I20260619 13:18:14.160029 17998 raft_consensus.cc:2781] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:14.160452 17998 raft_consensus.cc:686] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [term 1 LEADER]: Becoming Leader. State: Replica: 9801dd68b189432682801bd66638ce97, State: Running, Role: LEADER
I20260619 13:18:14.160964 17998 consensus_queue.cc:237] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b8b032e77f4dabbc10226a77ce185a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42883 } } peers { permanent_uuid: "dd1ba464cba745c0be8a12a38ca2136f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43531 } } peers { permanent_uuid: "9801dd68b189432682801bd66638ce97" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46239 } }
I20260619 13:18:14.165230 17514 catalog_manager.cc:5495] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9801dd68b189432682801bd66638ce97 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9801dd68b189432682801bd66638ce97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66b8b032e77f4dabbc10226a77ce185a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd1ba464cba745c0be8a12a38ca2136f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9801dd68b189432682801bd66638ce97" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46239 } health_report { overall_health: HEALTHY } } }
W20260619 13:18:14.195406 17973 tablet.cc:2369] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:18:14.213078 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:14.207664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.0.97.3:46239)
0619 13:18:14.207854 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:14.207860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:14.207900 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:18:14.208145 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:14.208151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:14.208247 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:14.208377 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:18:14.208756 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.209640 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.211052 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.211455 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.211577 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:14.212023 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:18:14.212524 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:18:14.212679 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:14.212732 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 13:18:14.212812 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:18:14.220949 17660 raft_consensus.cc:1261] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f [term 1 FOLLOWER]: Refusing update from remote peer 9801dd68b189432682801bd66638ce97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:18:14.220949 17794 raft_consensus.cc:1261] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a [term 1 FOLLOWER]: Refusing update from remote peer 9801dd68b189432682801bd66638ce97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:18:14.222009 18000 consensus_queue.cc:1035] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd1ba464cba745c0be8a12a38ca2136f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:18:14.222343 17998 consensus_queue.cc:1035] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66b8b032e77f4dabbc10226a77ce185a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:18:14.232602 18003 mvcc.cc:205] Tried to move back new op lower bound from 7298560385919320064 to 7298560385687339008. Current Snapshot: MvccSnapshot[applied={T|T < 7298560385919320064}]
I20260619 13:18:14.250407 17515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:18:14.250685 17515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:18:14.253958 17515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8fdab99704374d72b1d69a2fd5bcc340: Sending DeleteTablet for 3 replicas of tablet 01019109e8f44a358bd22df26fc5018d
I20260619 13:18:14.255069 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet 01019109e8f44a358bd22df26fc5018d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52920
I20260619 13:18:14.255371 17907 tablet_service.cc:1514] Processing DeleteTablet for tablet 01019109e8f44a358bd22df26fc5018d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59476
I20260619 13:18:14.255416 17640 tablet_service.cc:1514] Processing DeleteTablet for tablet 01019109e8f44a358bd22df26fc5018d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60228
I20260619 13:18:14.256911 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 17577
I20260619 13:18:14.266582 18015 tablet_replica.cc:330] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f: stopping tablet replica
I20260619 13:18:14.267560 18015 raft_consensus.cc:2229] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:18:14.268676 18015 raft_consensus.cc:2258] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:14.271749 18015 ts_tablet_manager.cc:1905] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:14.272110 18016 tablet_replica.cc:330] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97: stopping tablet replica
I20260619 13:18:14.272578 18016 raft_consensus.cc:2229] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:18:14.273262 18016 raft_consensus.cc:2258] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:14.279491 18014 tablet_replica.cc:330] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a: stopping tablet replica
I20260619 13:18:14.279874 18014 raft_consensus.cc:2229] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:18:14.280097 18016 ts_tablet_manager.cc:1905] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:14.280378 18014 raft_consensus.cc:2258] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:14.281894 18015 ts_tablet_manager.cc:1918] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:14.282047 18015 log.cc:1196] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/01019109e8f44a358bd22df26fc5018d
I20260619 13:18:14.282296 18014 ts_tablet_manager.cc:1905] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:14.282552 18015 ts_tablet_manager.cc:1939] T 01019109e8f44a358bd22df26fc5018d P dd1ba464cba745c0be8a12a38ca2136f: Deleting consensus metadata
I20260619 13:18:14.284547 17500 catalog_manager.cc:4841] TS dd1ba464cba745c0be8a12a38ca2136f (127.0.97.1:43531): tablet 01019109e8f44a358bd22df26fc5018d (table test-table [id=76d9ed1180ef4ac2b30dd0f423b62f88]) successfully deleted
Jun 19 13:18:14 dist-test-slave-mkt3 krb5kdc[17471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:18:14.287683 18016 ts_tablet_manager.cc:1918] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:14.287947 18016 log.cc:1196] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/01019109e8f44a358bd22df26fc5018d
I20260619 13:18:14.288534 18016 ts_tablet_manager.cc:1939] T 01019109e8f44a358bd22df26fc5018d P 9801dd68b189432682801bd66638ce97: Deleting consensus metadata
I20260619 13:18:14.290297 17500 catalog_manager.cc:4841] TS 9801dd68b189432682801bd66638ce97 (127.0.97.3:46239): tablet 01019109e8f44a358bd22df26fc5018d (table test-table [id=76d9ed1180ef4ac2b30dd0f423b62f88]) successfully deleted
I20260619 13:18:14.290557 18014 ts_tablet_manager.cc:1918] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:14.290710 18014 log.cc:1196] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/01019109e8f44a358bd22df26fc5018d
I20260619 13:18:14.291194 18014 ts_tablet_manager.cc:1939] T 01019109e8f44a358bd22df26fc5018d P 66b8b032e77f4dabbc10226a77ce185a: Deleting consensus metadata
I20260619 13:18:14.292788 17501 catalog_manager.cc:4841] TS 66b8b032e77f4dabbc10226a77ce185a (127.0.97.2:42883): tablet 01019109e8f44a358bd22df26fc5018d (table test-table [id=76d9ed1180ef4ac2b30dd0f423b62f88]) successfully deleted
I20260619 13:18:14.294335 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:14.258275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.0.97.1:43531)
0619 13:18:14.258449 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:14.258455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:14.272302 (+ 13847us) server_negotiation.cc:407] Connection header received
0619 13:18:14.272725 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:14.272731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:14.272819 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:14.272891 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:14.274554 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.275663 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.277573 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.278217 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.278328 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:14.288127 (+ 9799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:14.288143 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:14.288162 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:14.288205 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:14.291663 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:14.292307 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:14.292314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:14.292316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:14.292432 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:14.292823 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:14.292826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:14.292828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:14.293184 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:14.293462 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:14.293762 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 13:18:14.294008 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:18:14.294938 17700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:14.381059 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 17577
I20260619 13:18:14.395756 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 17711
Jun 19 13:18:14 dist-test-slave-mkt3 krb5kdc[17471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:18:14.410384 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:14.396558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52926 (local address 127.0.97.2:42883)
0619 13:18:14.396742 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:14.396748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:14.396769 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:18:14.396834 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:14.396840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:14.396918 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:14.396999 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:14.398291 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.399000 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.400391 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.400791 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.400896 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:14.404841 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:14.404855 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:14.404870 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:14.404903 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:14.407547 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:14.408216 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:14.408223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:14.408226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:14.408345 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:14.408750 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:14.408753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:14.408755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:14.409131 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:14.409411 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:14.409701 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 13:18:14.410017 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:18:14.411074 17834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:14.512918 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 17711
I20260619 13:18:14.533340 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 17844
Jun 19 13:18:14 dist-test-slave-mkt3 krb5kdc[17471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:18:14.550509 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:14.534240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.0.97.3:46239)
0619 13:18:14.534396 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:14.534400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:14.534418 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:18:14.534524 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:14.534527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:14.534595 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:14.534677 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:14.536356 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.537263 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.539070 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:14.539525 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:14.539631 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:14.544812 (+ 5181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:14.544824 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:14.544840 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:14.544880 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:14.547804 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:14.548452 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:14.548459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:14.548461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:14.548623 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:14.548964 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:14.548967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:14.548969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:14.549328 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:14.549621 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:14.549939 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 13:18:14.550153 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:18:14.551239 17967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:14.670810 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 17844
I20260619 13:18:14.686306 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 17484
I20260619 13:18:14.687398 17546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:14.796330 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 17484
2026-06-19T13:18:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (3992 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T13:18:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:14Z Disabled control of system clock
I20260619 13:18:14.862300 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38949
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:18:15.173736 18025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:15.174080 18025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:15.174207 18025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:15.181443 18025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:15.181547 18025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:15.181593 18025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:15.181670 18025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:15.181730 18025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:15.190935 18025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38949
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:15.192210 18025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:15.193938 18025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:15.205175 18030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:15.205197 18031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:15.205159 18033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:15.205999 18025 server_base.cc:1034] running on GCE node
I20260619 13:18:15.206765 18025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:15.208143 18025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:15.209365 18025 hybrid_clock.cc:648] HybridClock initialized: now 1781875095209313 us; error 46 us; skew 500 ppm
I20260619 13:18:15.209704 18025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:18:15.211908 18025 webserver.cc:460] Webserver started at http://127.0.97.62:39551/ using document root <none> and password file <none>
I20260619 13:18:15.212558 18025 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:15.212654 18025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:15.212949 18025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:15.215752 18025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "90e82f7951904fc3a005392ef17981de"
format_stamp: "Formatted at 2026-06-19 13:18:15 on dist-test-slave-mkt3"
server_key: "aac168a011a2809caf97c080e2e856c8"
server_key_iv: "b082915235b2f70ab4cac7aa8fed1ab8"
server_key_version: "encryptionkey@0"
I20260619 13:18:15.216547 18025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "90e82f7951904fc3a005392ef17981de"
format_stamp: "Formatted at 2026-06-19 13:18:15 on dist-test-slave-mkt3"
server_key: "aac168a011a2809caf97c080e2e856c8"
server_key_iv: "b082915235b2f70ab4cac7aa8fed1ab8"
server_key_version: "encryptionkey@0"
I20260619 13:18:15.221398 18025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260619 13:18:15.224713 18039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:15.226027 18025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:18:15.226207 18025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "90e82f7951904fc3a005392ef17981de"
format_stamp: "Formatted at 2026-06-19 13:18:15 on dist-test-slave-mkt3"
server_key: "aac168a011a2809caf97c080e2e856c8"
server_key_iv: "b082915235b2f70ab4cac7aa8fed1ab8"
server_key_version: "encryptionkey@0"
I20260619 13:18:15.226399 18025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:15.246915 18025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:15.247715 18025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:15.247975 18025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:15.264308 18025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38949
I20260619 13:18:15.264325 18090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38949 every 8 connection(s)
I20260619 13:18:15.265518 18025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:15.268976 18091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:15.271593 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 18025
I20260619 13:18:15.271775 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:15.272294 388 external_mini_cluster.cc:1428] Setting key 80eb428a3b88aab685bdeaaac8c27ce2
I20260619 13:18:15.279960 18091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de: Bootstrap starting.
I20260619 13:18:15.282960 18091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:15.283973 18091 log.cc:824] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:15.285169 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:15.274134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.0.97.62:38949)
0619 13:18:15.275032 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:15.275042 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:15.275079 (+ 37us) server_negotiation.cc:407] Connection header received
0619 13:18:15.276693 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:15.276714 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:15.277153 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:15.277516 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:15.278093 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:15.279298 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:15.280771 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:15.281458 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:15.281622 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:15.282749 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:15.282757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:15.282770 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:15.283310 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:15.283403 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:15.283424 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:15.283525 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:15.283749 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 13:18:15.284023 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":317,"threads_started":1}
I20260619 13:18:15.286846 18091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de: No bootstrap required, opened a new log
I20260619 13:18:15.294345 18091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e82f7951904fc3a005392ef17981de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38949 } }
I20260619 13:18:15.294683 18091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:15.294762 18091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e82f7951904fc3a005392ef17981de, State: Initialized, Role: FOLLOWER
I20260619 13:18:15.295257 18091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e82f7951904fc3a005392ef17981de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38949 } }
I20260619 13:18:15.295455 18091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:15.295576 18091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:15.295737 18091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:15.297940 18091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e82f7951904fc3a005392ef17981de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38949 } }
I20260619 13:18:15.298398 18091 leader_election.cc:302] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90e82f7951904fc3a005392ef17981de; no voters:
I20260619 13:18:15.298913 18091 leader_election.cc:288] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:15.299031 18096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:15.299937 18096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [term 1 LEADER]: Becoming Leader. State: Replica: 90e82f7951904fc3a005392ef17981de, State: Running, Role: LEADER
I20260619 13:18:15.300449 18096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e82f7951904fc3a005392ef17981de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38949 } }
I20260619 13:18:15.300873 18091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:15.302887 18098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90e82f7951904fc3a005392ef17981de. Latest consensus state: current_term: 1 leader_uuid: "90e82f7951904fc3a005392ef17981de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e82f7951904fc3a005392ef17981de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38949 } } }
I20260619 13:18:15.303244 18098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:15.303700 18097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90e82f7951904fc3a005392ef17981de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e82f7951904fc3a005392ef17981de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38949 } } }
I20260619 13:18:15.304005 18097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:15.304230 18103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:15.311322 18103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:15.321372 18103 catalog_manager.cc:1347] Generated new cluster ID: daf8467b0d194c5db12f18646fe8ba07
I20260619 13:18:15.321482 18103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:15.339295 18103 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:15.340505 18103 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:15.348865 18103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de: Generated new TSK 0
I20260619 13:18:15.349634 18103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:15.358106 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38949
--builtin_ntp_servers=127.0.97.20:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:18:15.700582 18116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:15.700839 18116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:15.700932 18116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:15.708077 18116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:15.708177 18116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:15.708281 18116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:18:15.717291 18116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:15.718531 18116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:15.720235 18116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:15.733124 18121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:15.733181 18124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:15.733320 18122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:15.734010 18116 server_base.cc:1034] running on GCE node
I20260619 13:18:15.734546 18116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:15.735442 18116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:15.736624 18116 hybrid_clock.cc:648] HybridClock initialized: now 1781875095736567 us; error 53 us; skew 500 ppm
I20260619 13:18:15.736922 18116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:18:15.739073 18116 webserver.cc:460] Webserver started at http://127.0.97.1:46027/ using document root <none> and password file <none>
I20260619 13:18:15.739713 18116 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:15.739809 18116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:15.740101 18116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:15.742935 18116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "e739932ac30d4b27a3382d3c13322070"
format_stamp: "Formatted at 2026-06-19 13:18:15 on dist-test-slave-mkt3"
server_key: "90d513b032a63a16cf4136a2d50118ba"
server_key_iv: "323c6046ec4bea0e5d2c349742aef2e5"
server_key_version: "encryptionkey@0"
I20260619 13:18:15.743672 18116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e739932ac30d4b27a3382d3c13322070"
format_stamp: "Formatted at 2026-06-19 13:18:15 on dist-test-slave-mkt3"
server_key: "90d513b032a63a16cf4136a2d50118ba"
server_key_iv: "323c6046ec4bea0e5d2c349742aef2e5"
server_key_version: "encryptionkey@0"
I20260619 13:18:15.748420 18116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:18:15.751668 18130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:15.752872 18116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 13:18:15.753012 18116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "e739932ac30d4b27a3382d3c13322070"
format_stamp: "Formatted at 2026-06-19 13:18:15 on dist-test-slave-mkt3"
server_key: "90d513b032a63a16cf4136a2d50118ba"
server_key_iv: "323c6046ec4bea0e5d2c349742aef2e5"
server_key_version: "encryptionkey@0"
I20260619 13:18:15.753194 18116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:15.764297 18116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:15.765012 18116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:15.765244 18116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:15.766144 18116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:15.767500 18116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:15.767590 18116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:15.767700 18116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:15.767766 18116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:15.794693 18116 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33715
I20260619 13:18:15.794744 18242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33715 every 8 connection(s)
I20260619 13:18:15.795908 18116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:18:15.802975 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 18116
I20260619 13:18:15.803124 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:18:15.803507 388 external_mini_cluster.cc:1428] Setting key baff399a188c103ce56b1c88ff2b3290
I20260619 13:18:15.806402 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38949
--builtin_ntp_servers=127.0.97.20:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:18:15.808076 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:15.798296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42085 (local address 127.0.97.62:38949)
0619 13:18:15.798730 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:15.798735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:15.799891 (+ 1156us) server_negotiation.cc:407] Connection header received
0619 13:18:15.801606 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:15.801611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:15.801681 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:15.801773 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:15.802842 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:15.803619 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:15.805340 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:15.805783 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:15.805912 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:15.807430 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:15.807434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:15.807438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:15.807455 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:15.807499 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:15.807510 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:15.807577 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:15.807787 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 13:18:15.807856 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":206,"threads_started":1}
I20260619 13:18:15.809319 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:15.798867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38949 (local address 127.0.97.1:42085)
0619 13:18:15.799735 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:15.799770 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:15.801374 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:15.801983 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:15.801992 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:15.802084 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:15.802688 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:15.802713 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:15.803763 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:15.803768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:15.805199 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:15.805224 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:15.806028 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:15.806037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:15.806337 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:15.807085 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:18:15.807089 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:15.807149 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:18:15.807252 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:15.807594 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:15.807604 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:15.807612 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 13:18:15.807787 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 13:18:15.808137 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":261,"threads_started":1}
I20260619 13:18:15.810752 18243 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38949
I20260619 13:18:15.811040 18243 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:15.811910 18243 heartbeater.cc:511] Master 127.0.97.62:38949 requested a full tablet report, sending...
I20260619 13:18:15.814052 18056 ts_manager.cc:194] Registered new tserver with Master: e739932ac30d4b27a3382d3c13322070 (127.0.97.1:33715)
I20260619 13:18:15.815606 18056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:42085
W20260619 13:18:16.111200 18247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:16.111474 18247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:16.111588 18247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:16.118700 18247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:16.118796 18247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:16.118948 18247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:18:16.127735 18247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:16.128984 18247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:16.130730 18247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:16.142745 18253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:16.142812 18255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:16.142834 18252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:16.143523 18247 server_base.cc:1034] running on GCE node
I20260619 13:18:16.144057 18247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:16.144919 18247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:16.146109 18247 hybrid_clock.cc:648] HybridClock initialized: now 1781875096146057 us; error 44 us; skew 500 ppm
I20260619 13:18:16.146420 18247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:18:16.148526 18247 webserver.cc:460] Webserver started at http://127.0.97.2:42857/ using document root <none> and password file <none>
I20260619 13:18:16.149147 18247 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:16.149250 18247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:16.149538 18247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:16.152247 18247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "b4823e2add6048cca29e01515e4c0c5b"
format_stamp: "Formatted at 2026-06-19 13:18:16 on dist-test-slave-mkt3"
server_key: "c412a88f86b3b70c55bd44bc83d30147"
server_key_iv: "7cb6ed94f26fd18da7b4cb2cbd9a9ca7"
server_key_version: "encryptionkey@0"
I20260619 13:18:16.152976 18247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b4823e2add6048cca29e01515e4c0c5b"
format_stamp: "Formatted at 2026-06-19 13:18:16 on dist-test-slave-mkt3"
server_key: "c412a88f86b3b70c55bd44bc83d30147"
server_key_iv: "7cb6ed94f26fd18da7b4cb2cbd9a9ca7"
server_key_version: "encryptionkey@0"
I20260619 13:18:16.157825 18247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:18:16.161075 18261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:16.162393 18247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:18:16.162537 18247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "b4823e2add6048cca29e01515e4c0c5b"
format_stamp: "Formatted at 2026-06-19 13:18:16 on dist-test-slave-mkt3"
server_key: "c412a88f86b3b70c55bd44bc83d30147"
server_key_iv: "7cb6ed94f26fd18da7b4cb2cbd9a9ca7"
server_key_version: "encryptionkey@0"
I20260619 13:18:16.162724 18247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:16.179215 18247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:16.179973 18247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:16.180223 18247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:16.181123 18247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:16.182495 18247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:16.182590 18247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:16.182693 18247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:16.182760 18247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:16.211254 18247 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35601
I20260619 13:18:16.211304 18373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35601 every 8 connection(s)
I20260619 13:18:16.212495 18247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:18:16.219095 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 18247
I20260619 13:18:16.219257 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:18:16.223778 388 external_mini_cluster.cc:1428] Setting key ee3882a5ac999d267f976e96a9f92b6d
I20260619 13:18:16.225085 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:16.214920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34809 (local address 127.0.97.62:38949)
0619 13:18:16.215083 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:16.215087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:16.216453 (+ 1366us) server_negotiation.cc:407] Connection header received
0619 13:18:16.218280 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:16.218287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:16.218374 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:16.218477 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:16.219631 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.220719 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.222295 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.222897 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.223028 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:16.224256 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:16.224263 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:16.224267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:16.224291 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:16.224325 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:16.224338 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:16.224407 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:16.224652 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 13:18:16.224769 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:18:16.226159 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:16.215451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38949 (local address 127.0.97.2:34809)
0619 13:18:16.216307 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:16.216342 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:16.218003 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:16.218700 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:16.218709 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:16.218811 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:16.219460 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:16.219479 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.220865 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.220869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:16.222149 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:16.222161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.223078 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.223082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:16.223285 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:16.223930 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:18:16.223933 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:16.223977 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:18:16.224058 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:16.224425 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:16.224434 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:16.224443 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 13:18:16.224620 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 13:18:16.225013 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":248,"threads_started":1}
I20260619 13:18:16.226419 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38949
--builtin_ntp_servers=127.0.97.20:40775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:18:16.227463 18374 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38949
I20260619 13:18:16.227775 18374 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:16.228610 18374 heartbeater.cc:511] Master 127.0.97.62:38949 requested a full tablet report, sending...
I20260619 13:18:16.230032 18056 ts_manager.cc:194] Registered new tserver with Master: b4823e2add6048cca29e01515e4c0c5b (127.0.97.2:35601)
I20260619 13:18:16.231024 18056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:34809
W20260619 13:18:16.539239 18377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:16.539484 18377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:16.539573 18377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:16.546480 18377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:16.546567 18377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:16.546679 18377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:18:16.555341 18377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:16.556449 18377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:16.558197 18377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:16.569965 18382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:16.570010 18385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:16.569942 18383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:16.570847 18377 server_base.cc:1034] running on GCE node
I20260619 13:18:16.571372 18377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:16.572158 18377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:16.573338 18377 hybrid_clock.cc:648] HybridClock initialized: now 1781875096573290 us; error 42 us; skew 500 ppm
I20260619 13:18:16.573582 18377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:18:16.575551 18377 webserver.cc:460] Webserver started at http://127.0.97.3:42725/ using document root <none> and password file <none>
I20260619 13:18:16.576145 18377 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:16.576229 18377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:16.576472 18377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:16.579069 18377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "ca6bc12c6c3649f48adcd4a68cc314c6"
format_stamp: "Formatted at 2026-06-19 13:18:16 on dist-test-slave-mkt3"
server_key: "9f902900d75b262033be69acfaf0cccf"
server_key_iv: "7efc866fdd00ec859fc98061fed8fc31"
server_key_version: "encryptionkey@0"
I20260619 13:18:16.579751 18377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ca6bc12c6c3649f48adcd4a68cc314c6"
format_stamp: "Formatted at 2026-06-19 13:18:16 on dist-test-slave-mkt3"
server_key: "9f902900d75b262033be69acfaf0cccf"
server_key_iv: "7efc866fdd00ec859fc98061fed8fc31"
server_key_version: "encryptionkey@0"
I20260619 13:18:16.584335 18377 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:18:16.587422 18391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:16.588616 18377 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 13:18:16.588755 18377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "ca6bc12c6c3649f48adcd4a68cc314c6"
format_stamp: "Formatted at 2026-06-19 13:18:16 on dist-test-slave-mkt3"
server_key: "9f902900d75b262033be69acfaf0cccf"
server_key_iv: "7efc866fdd00ec859fc98061fed8fc31"
server_key_version: "encryptionkey@0"
I20260619 13:18:16.588935 18377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:16.616233 18377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:16.617049 18377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:16.617302 18377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:16.618297 18377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:16.619742 18377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:16.619835 18377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:16.619936 18377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:16.620002 18377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:16.648898 18377 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34449
I20260619 13:18:16.648983 18503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34449 every 8 connection(s)
I20260619 13:18:16.650139 18377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:18:16.660089 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 18377
I20260619 13:18:16.660243 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:18:16.660781 388 external_mini_cluster.cc:1428] Setting key b5ba032afd710c0a19944386d0dae6e5
I20260619 13:18:16.661969 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:16.652597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43125 (local address 127.0.97.62:38949)
0619 13:18:16.652779 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:16.652784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:16.654234 (+ 1450us) server_negotiation.cc:407] Connection header received
0619 13:18:16.655929 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:16.655933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:16.656006 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:16.656096 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:16.657134 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.657909 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.659309 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.659710 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.659832 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:16.661121 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:16.661127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:16.661131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:16.661152 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:16.661192 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:16.661208 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:16.661292 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:16.661543 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 13:18:16.661641 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:18:16.663003 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:16.653114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38949 (local address 127.0.97.3:43125)
0619 13:18:16.654065 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:16.654101 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:16.655681 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:16.656304 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:16.656311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:16.656398 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:16.656968 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:16.656992 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.658006 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.658009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:16.659192 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:16.659203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.659878 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.659882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:16.660070 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:16.660783 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:18:16.660785 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:16.660836 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:18:16.660921 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:16.661308 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:16.661320 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:16.661329 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 13:18:16.661514 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 13:18:16.661873 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":285,"threads_started":1}
I20260619 13:18:16.664220 18504 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38949
I20260619 13:18:16.664469 18504 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:16.665129 18504 heartbeater.cc:511] Master 127.0.97.62:38949 requested a full tablet report, sending...
I20260619 13:18:16.666432 18056 ts_manager.cc:194] Registered new tserver with Master: ca6bc12c6c3649f48adcd4a68cc314c6 (127.0.97.3:34449)
I20260619 13:18:16.667229 18056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:43125
I20260619 13:18:16.675525 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:18:16.684834 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:16.679304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.0.97.62:38949)
0619 13:18:16.679451 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:16.679457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:16.679851 (+ 394us) server_negotiation.cc:407] Connection header received
0619 13:18:16.680013 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:16.680018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:16.680095 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:16.680172 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:16.680653 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.681359 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.682753 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.683149 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.683256 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:16.684191 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:16.684195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:16.684198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:16.684216 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:16.684247 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:16.684259 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:16.684332 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:16.684456 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 13:18:16.684540 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 13:18:16.688179 18056 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:18:16.695147 18056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:18:16.732609 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:16.707228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34449 (local address 127.0.0.1:55340)
0619 13:18:16.724237 (+ 17009us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:16.724257 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:16.724374 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:16.724987 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:16.724995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:16.725025 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:16.725304 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:16.725316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.726916 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.726921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:16.729378 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:16.729393 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.731700 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.731706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:16.731904 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:16.731954 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 13:18:16.732077 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 13:18:16.732266 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16904,"thread_start_us":15319,"threads_started":1}
I20260619 13:18:16.733516 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:16.706028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33715 (local address 127.0.0.1:43972)
0619 13:18:16.722003 (+ 15975us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:16.722041 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:16.722224 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:16.725362 (+ 3138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:16.725366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:16.725388 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:16.725881 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:16.725894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.727729 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.727734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:16.730432 (+ 2698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:16.730447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.732525 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.732531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:16.732927 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:16.732951 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 13:18:16.733019 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 13:18:16.733149 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15705,"mutex_wait_us":455,"spinlock_wait_cycles":1280,"thread_start_us":15561,"threads_started":1}
I20260619 13:18:16.733547 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:16.707758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.0.97.3:34449)
0619 13:18:16.723522 (+ 15764us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:16.723533 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:16.724340 (+ 807us) server_negotiation.cc:407] Connection header received
0619 13:18:16.724528 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:16.724536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:16.724762 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:16.724953 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:16.725533 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.726808 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.729598 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.731181 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.731315 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:16.731591 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:16.733047 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0619 13:18:16.733235 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15576,"thread_start_us":15416,"threads_started":1}
I20260619 13:18:16.733639 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:16.706209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.0.97.1:33715)
0619 13:18:16.724825 (+ 18616us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:16.724833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:16.724859 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:18:16.724950 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:16.724957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:16.725168 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:16.725362 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:16.726048 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.727565 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.730663 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.732330 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.732445 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:16.732737 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:16.733102 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 13:18:16.733270 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18475,"thread_start_us":18364,"threads_started":1}
I20260619 13:18:16.734063 18520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:16.706118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35601 (local address 127.0.0.1:50032)
0619 13:18:16.722828 (+ 16710us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:16.722846 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:16.722947 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:16.725396 (+ 2449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:16.725400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:16.725425 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:16.725820 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:16.725829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.727833 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.727840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:16.730309 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:16.730323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.733295 (+ 2972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.733302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:16.733484 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:16.733521 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 13:18:16.733585 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 13:18:16.733717 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16568,"mutex_wait_us":78,"thread_start_us":17868,"threads_started":1}
I20260619 13:18:16.735620 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:16.706642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.0.97.2:35601)
0619 13:18:16.723915 (+ 17273us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:16.723923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:16.723955 (+ 32us) server_negotiation.cc:407] Connection header received
0619 13:18:16.724061 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:16.724069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:16.725209 (+ 1140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:16.725946 (+ 737us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:16.726211 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.727692 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.730555 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:16.733096 (+ 2541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:16.734369 (+ 1273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:16.734596 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:16.734658 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 13:18:16.735314 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17127,"thread_start_us":17017,"threads_started":1}
I20260619 13:18:16.737336 18178 tablet_service.cc:1467] Processing CreateTablet for tablet ee556ebccf1a4cb99a754c8f97c70386 (DEFAULT_TABLE table=test-table [id=11a9c8ff488a4c009fa9640bb418f038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:16.737309 18439 tablet_service.cc:1467] Processing CreateTablet for tablet ee556ebccf1a4cb99a754c8f97c70386 (DEFAULT_TABLE table=test-table [id=11a9c8ff488a4c009fa9640bb418f038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:16.738592 18178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee556ebccf1a4cb99a754c8f97c70386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:16.738708 18439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee556ebccf1a4cb99a754c8f97c70386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:16.738942 18309 tablet_service.cc:1467] Processing CreateTablet for tablet ee556ebccf1a4cb99a754c8f97c70386 (DEFAULT_TABLE table=test-table [id=11a9c8ff488a4c009fa9640bb418f038]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:16.740067 18309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee556ebccf1a4cb99a754c8f97c70386. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:16.749254 18522 tablet_bootstrap.cc:492] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070: Bootstrap starting.
I20260619 13:18:16.752068 18522 tablet_bootstrap.cc:654] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:16.752997 18523 tablet_bootstrap.cc:492] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6: Bootstrap starting.
I20260619 13:18:16.754382 18524 tablet_bootstrap.cc:492] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b: Bootstrap starting.
I20260619 13:18:16.754971 18522 log.cc:824] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:16.756433 18523 tablet_bootstrap.cc:654] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:16.757283 18524 tablet_bootstrap.cc:654] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:16.757565 18523 log.cc:824] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:16.758127 18522 tablet_bootstrap.cc:492] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070: No bootstrap required, opened a new log
I20260619 13:18:16.758245 18524 log.cc:824] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:16.758378 18522 ts_tablet_manager.cc:1397] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 13:18:16.760433 18523 tablet_bootstrap.cc:492] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6: No bootstrap required, opened a new log
I20260619 13:18:16.760680 18523 ts_tablet_manager.cc:1397] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 13:18:16.766727 18524 tablet_bootstrap.cc:492] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b: No bootstrap required, opened a new log
I20260619 13:18:16.766990 18524 ts_tablet_manager.cc:1397] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.002s
I20260619 13:18:16.768528 18522 raft_consensus.cc:348] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e739932ac30d4b27a3382d3c13322070" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33715 } } peers { permanent_uuid: "b4823e2add6048cca29e01515e4c0c5b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "ca6bc12c6c3649f48adcd4a68cc314c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34449 } }
I20260619 13:18:16.768960 18522 raft_consensus.cc:374] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:16.769063 18522 raft_consensus.cc:729] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e739932ac30d4b27a3382d3c13322070, State: Initialized, Role: FOLLOWER
I20260619 13:18:16.769663 18522 consensus_queue.cc:260] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e739932ac30d4b27a3382d3c13322070" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33715 } } peers { permanent_uuid: "b4823e2add6048cca29e01515e4c0c5b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "ca6bc12c6c3649f48adcd4a68cc314c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34449 } }
I20260619 13:18:16.770781 18243 heartbeater.cc:503] Master 127.0.97.62:38949 was elected leader, sending a full tablet report...
I20260619 13:18:16.770614 18523 raft_consensus.cc:348] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e739932ac30d4b27a3382d3c13322070" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33715 } } peers { permanent_uuid: "b4823e2add6048cca29e01515e4c0c5b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "ca6bc12c6c3649f48adcd4a68cc314c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34449 } }
I20260619 13:18:16.771004 18523 raft_consensus.cc:374] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:16.771101 18523 raft_consensus.cc:729] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6bc12c6c3649f48adcd4a68cc314c6, State: Initialized, Role: FOLLOWER
I20260619 13:18:16.771665 18523 consensus_queue.cc:260] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e739932ac30d4b27a3382d3c13322070" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33715 } } peers { permanent_uuid: "b4823e2add6048cca29e01515e4c0c5b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "ca6bc12c6c3649f48adcd4a68cc314c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34449 } }
I20260619 13:18:16.774724 18524 raft_consensus.cc:348] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e739932ac30d4b27a3382d3c13322070" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33715 } } peers { permanent_uuid: "b4823e2add6048cca29e01515e4c0c5b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "ca6bc12c6c3649f48adcd4a68cc314c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34449 } }
I20260619 13:18:16.775108 18524 raft_consensus.cc:374] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:16.775197 18524 raft_consensus.cc:729] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4823e2add6048cca29e01515e4c0c5b, State: Initialized, Role: FOLLOWER
I20260619 13:18:16.775713 18524 consensus_queue.cc:260] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e739932ac30d4b27a3382d3c13322070" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33715 } } peers { permanent_uuid: "b4823e2add6048cca29e01515e4c0c5b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "ca6bc12c6c3649f48adcd4a68cc314c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34449 } }
I20260619 13:18:16.775880 18522 ts_tablet_manager.cc:1428] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070: Time spent starting tablet: real 0.017s user 0.009s sys 0.004s
I20260619 13:18:16.776762 18374 heartbeater.cc:503] Master 127.0.97.62:38949 was elected leader, sending a full tablet report...
I20260619 13:18:16.777297 18524 ts_tablet_manager.cc:1428] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260619 13:18:16.777881 18504 heartbeater.cc:503] Master 127.0.97.62:38949 was elected leader, sending a full tablet report...
I20260619 13:18:16.778822 18523 ts_tablet_manager.cc:1428] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6: Time spent starting tablet: real 0.018s user 0.010s sys 0.008s
W20260619 13:18:16.797969 18244 tablet.cc:2369] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:18:16.901641 18505 tablet.cc:2369] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:18:16.964335 18375 tablet.cc:2369] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:18:17.133318 18528 raft_consensus.cc:482] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:18:17.133622 18528 raft_consensus.cc:504] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e739932ac30d4b27a3382d3c13322070" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33715 } } peers { permanent_uuid: "b4823e2add6048cca29e01515e4c0c5b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "ca6bc12c6c3649f48adcd4a68cc314c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34449 } }
I20260619 13:18:17.135334 18528 leader_election.cc:288] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4823e2add6048cca29e01515e4c0c5b (127.0.97.2:35601), ca6bc12c6c3649f48adcd4a68cc314c6 (127.0.97.3:34449)
I20260619 13:18:17.141508 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:17.136352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41997 (local address 127.0.97.3:34449)
0619 13:18:17.136532 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:17.136538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:17.136564 (+ 26us) server_negotiation.cc:407] Connection header received
0619 13:18:17.136678 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:17.136684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:17.136747 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:17.136815 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:17.137369 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.138109 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.139954 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.140677 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.140781 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:17.140871 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:17.141170 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 13:18:17.141284 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:18:17.141608 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:17.135728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59615 (local address 127.0.97.2:35601)
0619 13:18:17.135933 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:17.135939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:17.136321 (+ 382us) server_negotiation.cc:407] Connection header received
0619 13:18:17.136577 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:17.136583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:17.136659 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:17.136726 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:17.137427 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.138163 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.140017 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.140780 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.140885 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:17.140968 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:17.141251 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 13:18:17.141370 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:18:17.141636 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:17.135717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34449 (local address 127.0.97.1:41997)
0619 13:18:17.136419 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:17.136435 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:17.136561 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:17.136841 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:17.136845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:17.136868 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:17.137201 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:17.137211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.138236 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.138240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:17.139781 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:17.139794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.140834 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.140840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:17.141007 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:17.141046 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 13:18:17.141129 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 13:18:17.141255 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":95,"thread_start_us":672,"threads_started":1}
I20260619 13:18:17.141721 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:17.135535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35601 (local address 127.0.97.1:59615)
0619 13:18:17.136186 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:17.136211 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:17.136375 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:17.136886 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:17.136889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:17.136907 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:17.137292 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:17.137301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.138297 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.138301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:17.139859 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:17.139871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.140947 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.140953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:17.141112 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:17.141134 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:18:17.141209 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 13:18:17.141324 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":43,"thread_start_us":302,"threads_started":1}
I20260619 13:18:17.142714 18459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee556ebccf1a4cb99a754c8f97c70386" candidate_uuid: "e739932ac30d4b27a3382d3c13322070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca6bc12c6c3649f48adcd4a68cc314c6" is_pre_election: true
I20260619 13:18:17.142743 18329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee556ebccf1a4cb99a754c8f97c70386" candidate_uuid: "e739932ac30d4b27a3382d3c13322070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4823e2add6048cca29e01515e4c0c5b" is_pre_election: true
I20260619 13:18:17.143164 18459 raft_consensus.cc:2454] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e739932ac30d4b27a3382d3c13322070 in term 0.
I20260619 13:18:17.143173 18329 raft_consensus.cc:2454] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e739932ac30d4b27a3382d3c13322070 in term 0.
I20260619 13:18:17.143725 18134 leader_election.cc:302] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4823e2add6048cca29e01515e4c0c5b, e739932ac30d4b27a3382d3c13322070; no voters:
I20260619 13:18:17.144088 18528 raft_consensus.cc:2781] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:18:17.144245 18528 raft_consensus.cc:482] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:18:17.144337 18528 raft_consensus.cc:3037] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:17.146414 18528 raft_consensus.cc:504] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e739932ac30d4b27a3382d3c13322070" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33715 } } peers { permanent_uuid: "b4823e2add6048cca29e01515e4c0c5b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "ca6bc12c6c3649f48adcd4a68cc314c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34449 } }
I20260619 13:18:17.146973 18528 leader_election.cc:288] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [CANDIDATE]: Term 1 election: Requested vote from peers b4823e2add6048cca29e01515e4c0c5b (127.0.97.2:35601), ca6bc12c6c3649f48adcd4a68cc314c6 (127.0.97.3:34449)
I20260619 13:18:17.147356 18329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee556ebccf1a4cb99a754c8f97c70386" candidate_uuid: "e739932ac30d4b27a3382d3c13322070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4823e2add6048cca29e01515e4c0c5b"
I20260619 13:18:17.147403 18459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee556ebccf1a4cb99a754c8f97c70386" candidate_uuid: "e739932ac30d4b27a3382d3c13322070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca6bc12c6c3649f48adcd4a68cc314c6"
I20260619 13:18:17.147529 18329 raft_consensus.cc:3037] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:17.147558 18459 raft_consensus.cc:3037] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:17.149610 18459 raft_consensus.cc:2454] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e739932ac30d4b27a3382d3c13322070 in term 1.
I20260619 13:18:17.149616 18329 raft_consensus.cc:2454] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e739932ac30d4b27a3382d3c13322070 in term 1.
I20260619 13:18:17.150002 18131 leader_election.cc:302] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b4823e2add6048cca29e01515e4c0c5b, ca6bc12c6c3649f48adcd4a68cc314c6, e739932ac30d4b27a3382d3c13322070; no voters:
I20260619 13:18:17.150307 18528 raft_consensus.cc:2781] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:17.150727 18528 raft_consensus.cc:686] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [term 1 LEADER]: Becoming Leader. State: Replica: e739932ac30d4b27a3382d3c13322070, State: Running, Role: LEADER
I20260619 13:18:17.151165 18528 consensus_queue.cc:237] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e739932ac30d4b27a3382d3c13322070" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33715 } } peers { permanent_uuid: "b4823e2add6048cca29e01515e4c0c5b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } } peers { permanent_uuid: "ca6bc12c6c3649f48adcd4a68cc314c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34449 } }
I20260619 13:18:17.156011 18056 catalog_manager.cc:5495] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 reported cstate change: term changed from 0 to 1, leader changed from <none> to e739932ac30d4b27a3382d3c13322070 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e739932ac30d4b27a3382d3c13322070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e739932ac30d4b27a3382d3c13322070" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4823e2add6048cca29e01515e4c0c5b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca6bc12c6c3649f48adcd4a68cc314c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34449 } health_report { overall_health: UNKNOWN } } }
I20260619 13:18:17.184587 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:17.179386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.0.97.1:33715)
0619 13:18:17.179592 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:17.179597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:17.179614 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:18:17.179791 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:17.179795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:17.179881 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:17.180002 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:18:17.180352 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.181061 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.182622 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.183024 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.183131 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:17.183529 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:18:17.184038 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:18:17.184152 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:17.184237 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 13:18:17.184301 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:18:17.192632 18459 raft_consensus.cc:1261] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6 [term 1 FOLLOWER]: Refusing update from remote peer e739932ac30d4b27a3382d3c13322070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:18:17.192658 18329 raft_consensus.cc:1261] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b [term 1 FOLLOWER]: Refusing update from remote peer e739932ac30d4b27a3382d3c13322070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:18:17.193499 18533 consensus_queue.cc:1035] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca6bc12c6c3649f48adcd4a68cc314c6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:18:17.193830 18528 consensus_queue.cc:1035] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4823e2add6048cca29e01515e4c0c5b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:18:17.216405 18056 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:18:17.216645 18056 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:18:17.219871 18056 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90e82f7951904fc3a005392ef17981de: Sending DeleteTablet for 3 replicas of tablet ee556ebccf1a4cb99a754c8f97c70386
I20260619 13:18:17.220860 18178 tablet_service.cc:1514] Processing DeleteTablet for tablet ee556ebccf1a4cb99a754c8f97c70386 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:17 UTC) from {username='slave'} at 127.0.0.1:43972
I20260619 13:18:17.221552 18309 tablet_service.cc:1514] Processing DeleteTablet for tablet ee556ebccf1a4cb99a754c8f97c70386 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:17 UTC) from {username='slave'} at 127.0.0.1:50032
I20260619 13:18:17.221657 18439 tablet_service.cc:1514] Processing DeleteTablet for tablet ee556ebccf1a4cb99a754c8f97c70386 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:17 UTC) from {username='slave'} at 127.0.0.1:55340
I20260619 13:18:17.224186 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 18116
I20260619 13:18:17.235169 18547 tablet_replica.cc:330] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b: stopping tablet replica
I20260619 13:18:17.235620 18547 raft_consensus.cc:2229] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:18:17.236155 18547 raft_consensus.cc:2258] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:17.236869 18546 tablet_replica.cc:330] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070: stopping tablet replica
I20260619 13:18:17.237264 18546 raft_consensus.cc:2229] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:18:17.238111 18546 raft_consensus.cc:2258] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:17.239089 18547 ts_tablet_manager.cc:1905] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:17.241235 18546 ts_tablet_manager.cc:1905] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:17.245077 18548 tablet_replica.cc:330] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6: stopping tablet replica
I20260619 13:18:17.245801 18547 ts_tablet_manager.cc:1918] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:17.246054 18547 log.cc:1196] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/ee556ebccf1a4cb99a754c8f97c70386
I20260619 13:18:17.246605 18547 ts_tablet_manager.cc:1939] T ee556ebccf1a4cb99a754c8f97c70386 P b4823e2add6048cca29e01515e4c0c5b: Deleting consensus metadata
I20260619 13:18:17.247499 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:17.226754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.0.97.1:33715)
0619 13:18:17.227234 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:17.227240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:17.239142 (+ 11902us) server_negotiation.cc:407] Connection header received
0619 13:18:17.239352 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:17.239357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:17.239460 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:17.239586 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:17.240063 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.241266 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.243272 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.243945 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.244048 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:17.245634 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:17.245645 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:17.245666 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:17.246592 (+ 926us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:17.246660 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:17.246689 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:17.246770 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:17.247023 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 13:18:17.247136 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330}
I20260619 13:18:17.248445 18043 catalog_manager.cc:4841] TS b4823e2add6048cca29e01515e4c0c5b (127.0.97.2:35601): tablet ee556ebccf1a4cb99a754c8f97c70386 (table test-table [id=11a9c8ff488a4c009fa9640bb418f038]) successfully deleted
I20260619 13:18:17.248652 18546 ts_tablet_manager.cc:1918] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:17.248790 18238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:17.248795 18546 log.cc:1196] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/ee556ebccf1a4cb99a754c8f97c70386
I20260619 13:18:17.249074 18548 raft_consensus.cc:2229] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:18:17.249612 18548 raft_consensus.cc:2258] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:17.251358 18548 ts_tablet_manager.cc:1905] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:17.256685 18548 ts_tablet_manager.cc:1918] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:17.256824 18548 log.cc:1196] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/ee556ebccf1a4cb99a754c8f97c70386
I20260619 13:18:17.257207 18548 ts_tablet_manager.cc:1939] T ee556ebccf1a4cb99a754c8f97c70386 P ca6bc12c6c3649f48adcd4a68cc314c6: Deleting consensus metadata
I20260619 13:18:17.258589 18040 catalog_manager.cc:4841] TS ca6bc12c6c3649f48adcd4a68cc314c6 (127.0.97.3:34449): tablet ee556ebccf1a4cb99a754c8f97c70386 (table test-table [id=11a9c8ff488a4c009fa9640bb418f038]) successfully deleted
I20260619 13:18:17.346151 18546 ts_tablet_manager.cc:1939] T ee556ebccf1a4cb99a754c8f97c70386 P e739932ac30d4b27a3382d3c13322070: Deleting consensus metadata
I20260619 13:18:17.346575 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 18116
W20260619 13:18:17.361371 18042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33715 (error 108)
I20260619 13:18:17.361541 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 18247
I20260619 13:18:17.363175 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:17.362446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33715 (local address 127.0.0.1:44012)
0619 13:18:17.362862 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:17.362976 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":207,"threads_started":1}
W20260619 13:18:17.363404 18042 catalog_manager.cc:4584] TS e739932ac30d4b27a3382d3c13322070 (127.0.97.1:33715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee556ebccf1a4cb99a754c8f97c70386: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33715: connect: Connection refused (error 111)
I20260619 13:18:17.368368 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:17.362322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.0.97.2:35601)
0619 13:18:17.362511 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:17.362518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:17.362572 (+ 54us) server_negotiation.cc:407] Connection header received
0619 13:18:17.362778 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:17.362782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:17.362876 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:17.362964 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:17.363443 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.364168 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.365476 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.365954 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.366062 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:17.367106 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:17.367115 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:17.367128 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:17.367680 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:17.367726 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:17.367745 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:17.367843 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:17.367962 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 13:18:17.368059 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:18:17.368884 18369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:17.416890 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:17.416422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33715 (local address 127.0.0.1:44028)
0619 13:18:17.416644 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:17.416741 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 13:18:17.417937 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:17.417541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33715 (local address 127.0.0.1:44036)
0619 13:18:17.417669 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:17.417751 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 13:18:17.455228 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 18247
I20260619 13:18:17.469120 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 18377
I20260619 13:18:17.475397 18521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:17.469827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.0.97.3:34449)
0619 13:18:17.470006 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:17.470010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:17.470029 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:18:17.470165 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:17.470169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:17.470269 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:17.470352 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:17.470727 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.471427 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.472727 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:17.473133 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:17.473241 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:17.474156 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:17.474165 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:17.474179 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:17.474752 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:17.474800 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:17.474820 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:17.474909 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:17.475000 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 13:18:17.475088 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 13:18:17.475948 18499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:17.479123 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:17.478622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33715 (local address 127.0.0.1:44050)
0619 13:18:17.478868 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:17.478953 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260619 13:18:17.480098 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:17.479790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33715 (local address 127.0.0.1:44052)
0619 13:18:17.479908 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:17.479978 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 13:18:17.560891 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:17.560260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33715 (local address 127.0.0.1:44058)
0619 13:18:17.560596 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:17.560710 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260619 13:18:17.562039 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:17.561546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33715 (local address 127.0.0.1:44070)
0619 13:18:17.561759 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:17.561872 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 13:18:17.562045 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 18377
I20260619 13:18:17.574576 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 18025
I20260619 13:18:17.575558 18086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:17.681804 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 18025
2026-06-19T13:18:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2883 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:17 dist-test-slave-mkt3 krb5kdc[18558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:17 dist-test-slave-mkt3 krb5kdc[18558](info): set up 2 sockets
Jun 19 13:18:17 dist-test-slave-mkt3 krb5kdc[18558](info): commencing operation
krb5kdc: starting...
W20260619 13:18:18.697575 388 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:18 dist-test-slave-mkt3 krb5kdc[18558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:18:18.921198 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37163
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:19.242671 18571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:19.242957 18571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:19.243094 18571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:19.250221 18571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:19.250329 18571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:19.250401 18571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:19.250474 18571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:19.250536 18571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:19.260073 18571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:19.261308 18571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:19.262605 18571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:19.273288 18576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:19.273334 18577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:19.273254 18579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:19.274354 18571 server_base.cc:1034] running on GCE node
I20260619 13:18:19.275094 18571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:19.276415 18571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:19.277623 18571 hybrid_clock.cc:648] HybridClock initialized: now 1781875099277578 us; error 33 us; skew 500 ppm
I20260619 13:18:19.277961 18571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:19 dist-test-slave-mkt3 krb5kdc[18558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:19.280840 18571 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:18:19.282271 18571 webserver.cc:460] Webserver started at http://127.0.97.62:41769/ using document root <none> and password file <none>
I20260619 13:18:19.282891 18571 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:19.282984 18571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:19.283227 18571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:19.286024 18571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "5fc91a310735467383fe049ed33b0514"
format_stamp: "Formatted at 2026-06-19 13:18:19 on dist-test-slave-mkt3"
server_key: "d21c6bb25671d0e8bb635a045707d4d8"
server_key_iv: "f2cc5c7097b5284ae5803f99c05cb081"
server_key_version: "encryptionkey@0"
I20260619 13:18:19.286693 18571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "5fc91a310735467383fe049ed33b0514"
format_stamp: "Formatted at 2026-06-19 13:18:19 on dist-test-slave-mkt3"
server_key: "d21c6bb25671d0e8bb635a045707d4d8"
server_key_iv: "f2cc5c7097b5284ae5803f99c05cb081"
server_key_version: "encryptionkey@0"
I20260619 13:18:19.291476 18571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:18:19.294730 18586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:19.295995 18571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:18:19.296151 18571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "5fc91a310735467383fe049ed33b0514"
format_stamp: "Formatted at 2026-06-19 13:18:19 on dist-test-slave-mkt3"
server_key: "d21c6bb25671d0e8bb635a045707d4d8"
server_key_iv: "f2cc5c7097b5284ae5803f99c05cb081"
server_key_version: "encryptionkey@0"
I20260619 13:18:19.296325 18571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:19.332191 18571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:19.336267 18571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:19.336529 18571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:19.353371 18571 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37163
I20260619 13:18:19.353411 18637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37163 every 8 connection(s)
I20260619 13:18:19.354682 18571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:19.358251 18638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:19.362438 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 18571
I20260619 13:18:19.362592 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:19.362986 388 external_mini_cluster.cc:1428] Setting key f83641987c5bfac29149702e7d2dfef2
I20260619 13:18:19.368932 18638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514: Bootstrap starting.
I20260619 13:18:19.372305 18638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:19.373406 18638 log.cc:824] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:19.376449 18638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514: No bootstrap required, opened a new log
Jun 19 13:18:19 dist-test-slave-mkt3 krb5kdc[18558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:18:19.383809 18638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc91a310735467383fe049ed33b0514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37163 } }
I20260619 13:18:19.384205 18638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:19.384296 18638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fc91a310735467383fe049ed33b0514, State: Initialized, Role: FOLLOWER
I20260619 13:18:19.384862 18638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc91a310735467383fe049ed33b0514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37163 } }
I20260619 13:18:19.385066 18641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:19.364677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55822 (local address 127.0.97.62:37163)
0619 13:18:19.365414 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:19.365426 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:19.365465 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:18:19.367559 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:19.367581 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:19.367968 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:19.368273 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:19.369974 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:19.371282 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:19.372984 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:19.373665 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:19.373807 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:19.378264 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:19.378272 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:19.378285 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:19.378323 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:19.381290 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:19.381925 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:19.381932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:19.381937 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:19.382095 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:19.382441 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:19.382445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:19.382447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:19.382951 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:19.383232 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:19.383654 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 13:18:19.384049 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":323,"threads_started":1}
I20260619 13:18:19.385094 18638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:19.385222 18638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:19.385383 18638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:19.387780 18638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc91a310735467383fe049ed33b0514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37163 } }
I20260619 13:18:19.388216 18638 leader_election.cc:302] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fc91a310735467383fe049ed33b0514; no voters:
I20260619 13:18:19.388712 18638 leader_election.cc:288] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:19.388814 18643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:19.389413 18643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [term 1 LEADER]: Becoming Leader. State: Replica: 5fc91a310735467383fe049ed33b0514, State: Running, Role: LEADER
I20260619 13:18:19.389869 18643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc91a310735467383fe049ed33b0514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37163 } }
I20260619 13:18:19.390674 18638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:19.392416 18645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fc91a310735467383fe049ed33b0514. Latest consensus state: current_term: 1 leader_uuid: "5fc91a310735467383fe049ed33b0514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc91a310735467383fe049ed33b0514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37163 } } }
I20260619 13:18:19.392803 18645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:19.393563 18644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fc91a310735467383fe049ed33b0514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fc91a310735467383fe049ed33b0514" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37163 } } }
I20260619 13:18:19.393952 18644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:19.398928 18649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:19.403153 18649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:19.411124 18649 catalog_manager.cc:1347] Generated new cluster ID: 50493e7f0ba14b779ddf05eecf2e8b81
I20260619 13:18:19.411266 18649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:19.421958 18649 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:19.423725 18649 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:19.430330 18649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514: Generated new TSK 0
I20260619 13:18:19.431392 18649 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:18:19.485659 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37163
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:19.798151 18664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:19.798413 18664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:19.798547 18664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:19.805729 18664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:19.805873 18664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:19.806026 18664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:18:19.815374 18664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:19.816570 18664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:19.817833 18664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:19.829753 18669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:19.829749 18670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:19.829792 18672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:19.830812 18664 server_base.cc:1034] running on GCE node
I20260619 13:18:19.831354 18664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:19.832213 18664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:19.833441 18664 hybrid_clock.cc:648] HybridClock initialized: now 1781875099833389 us; error 42 us; skew 500 ppm
I20260619 13:18:19.833752 18664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:19 dist-test-slave-mkt3 krb5kdc[18558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:19.836668 18664 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:18:19.838079 18664 webserver.cc:460] Webserver started at http://127.0.97.1:37767/ using document root <none> and password file <none>
I20260619 13:18:19.838721 18664 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:19.838829 18664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:19.839099 18664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:19.841874 18664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "1a20a2848dfb4c94826b0eb5a914e380"
format_stamp: "Formatted at 2026-06-19 13:18:19 on dist-test-slave-mkt3"
server_key: "a3c1ae8a27ec705161614697fe53b179"
server_key_iv: "07c1c9262bbfea7e118b3f1e742fe5db"
server_key_version: "encryptionkey@0"
I20260619 13:18:19.842617 18664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1a20a2848dfb4c94826b0eb5a914e380"
format_stamp: "Formatted at 2026-06-19 13:18:19 on dist-test-slave-mkt3"
server_key: "a3c1ae8a27ec705161614697fe53b179"
server_key_iv: "07c1c9262bbfea7e118b3f1e742fe5db"
server_key_version: "encryptionkey@0"
I20260619 13:18:19.847313 18664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260619 13:18:19.850615 18679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:19.851874 18664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:18:19.852015 18664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "1a20a2848dfb4c94826b0eb5a914e380"
format_stamp: "Formatted at 2026-06-19 13:18:19 on dist-test-slave-mkt3"
server_key: "a3c1ae8a27ec705161614697fe53b179"
server_key_iv: "07c1c9262bbfea7e118b3f1e742fe5db"
server_key_version: "encryptionkey@0"
I20260619 13:18:19.852201 18664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:19.879047 18664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:19.882740 18664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:19.883008 18664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:19.883888 18664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:19.885134 18664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:19.885208 18664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:19.885321 18664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:19.885371 18664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:19.914337 18664 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37623
I20260619 13:18:19.914402 18791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37623 every 8 connection(s)
I20260619 13:18:19.915526 18664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:18:19.918740 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 18664
I20260619 13:18:19.918896 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:18:19.919330 388 external_mini_cluster.cc:1428] Setting key 89eb84a00dc65a7b4b4b6cbdd4799b53
Jun 19 13:18:19 dist-test-slave-mkt3 krb5kdc[18558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:18:19.935353 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:19.917957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36821 (local address 127.0.97.62:37163)
0619 13:18:19.918377 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:19.918382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:19.919345 (+ 963us) server_negotiation.cc:407] Connection header received
0619 13:18:19.921561 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:19.921568 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:19.921656 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:19.921785 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:19.923172 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:19.924164 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:19.925634 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:19.926250 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:19.926368 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:19.929815 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:19.929822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:19.929827 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:19.929886 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:19.932327 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:19.933062 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:19.933068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:19.933070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:19.933215 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:19.933625 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:19.933629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:19.933631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:19.933955 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:19.934090 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:19.934765 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 13:18:19.934995 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":231,"threads_started":1}
I20260619 13:18:19.936318 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:19.918420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37163 (local address 127.0.97.1:36821)
0619 13:18:19.919164 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:19.919215 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:19.921307 (+ 2092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:19.921978 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:19.921986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:19.922467 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:19.923011 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:19.923028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:19.924295 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:19.924299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:19.925500 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:19.925511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:19.926436 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:19.926441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:19.926647 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:19.927254 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:18:19.927275 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:19.929597 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:19.932469 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:19.932478 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:19.932495 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:19.932955 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:19.933319 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:19.933325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:19.933329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:19.933512 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:19.934082 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:19.934090 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:19.934425 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 13:18:19.934661 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 13:18:19.935203 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":293,"threads_started":1}
I20260619 13:18:19.937928 18792 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37163
I20260619 13:18:19.938228 18792 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:19.938910 18792 heartbeater.cc:511] Master 127.0.97.62:37163 requested a full tablet report, sending...
I20260619 13:18:19.941155 18603 ts_manager.cc:194] Registered new tserver with Master: 1a20a2848dfb4c94826b0eb5a914e380 (127.0.97.1:37623)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 13:18:19.942797 18603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:18:19.968122 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37163
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:20.276707 18798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:20.276955 18798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:20.277068 18798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:20.284031 18798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:20.284129 18798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:20.284271 18798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:18:20.293675 18798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:20.294875 18798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:20.296114 18798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:20.308359 18803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:20.308386 18804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:20.308490 18806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:20.308864 18798 server_base.cc:1034] running on GCE node
I20260619 13:18:20.309449 18798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:20.310351 18798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:20.311582 18798 hybrid_clock.cc:648] HybridClock initialized: now 1781875100311510 us; error 40 us; skew 500 ppm
I20260619 13:18:20.311892 18798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:20 dist-test-slave-mkt3 krb5kdc[18558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:20.314983 18798 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:18:20.316324 18798 webserver.cc:460] Webserver started at http://127.0.97.2:40045/ using document root <none> and password file <none>
I20260619 13:18:20.316948 18798 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:20.317061 18798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:20.317341 18798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:20.320132 18798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "0c158d085cff4d38bcd2cae52afb9efb"
format_stamp: "Formatted at 2026-06-19 13:18:20 on dist-test-slave-mkt3"
server_key: "0ab9ab81479dbb46562b387b58635aca"
server_key_iv: "c388edbb3e1e4b057cc2c7624ffd1961"
server_key_version: "encryptionkey@0"
I20260619 13:18:20.320847 18798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0c158d085cff4d38bcd2cae52afb9efb"
format_stamp: "Formatted at 2026-06-19 13:18:20 on dist-test-slave-mkt3"
server_key: "0ab9ab81479dbb46562b387b58635aca"
server_key_iv: "c388edbb3e1e4b057cc2c7624ffd1961"
server_key_version: "encryptionkey@0"
I20260619 13:18:20.325536 18798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260619 13:18:20.328866 18813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:20.330137 18798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:18:20.330271 18798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "0c158d085cff4d38bcd2cae52afb9efb"
format_stamp: "Formatted at 2026-06-19 13:18:20 on dist-test-slave-mkt3"
server_key: "0ab9ab81479dbb46562b387b58635aca"
server_key_iv: "c388edbb3e1e4b057cc2c7624ffd1961"
server_key_version: "encryptionkey@0"
I20260619 13:18:20.330435 18798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:20.343334 18798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:20.347103 18798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:20.347345 18798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:20.348193 18798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:20.349375 18798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:20.349447 18798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:20.349558 18798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:20.349613 18798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:20.378795 18798 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36821
I20260619 13:18:20.378837 18925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36821 every 8 connection(s)
I20260619 13:18:20.379940 18798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:18:20.388484 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 18798
I20260619 13:18:20.388689 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:18:20.389219 388 external_mini_cluster.cc:1428] Setting key 209381ab6db7916c7c011251724970e0
Jun 19 13:18:20 dist-test-slave-mkt3 krb5kdc[18558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:18:20.399240 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:20.382298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47455 (local address 127.0.97.62:37163)
0619 13:18:20.382454 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:20.382459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:20.383589 (+ 1130us) server_negotiation.cc:407] Connection header received
0619 13:18:20.385336 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:20.385343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:20.385424 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:20.385509 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:20.386965 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.387685 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.389742 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.390169 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.390294 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:20.393752 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:20.393758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:20.393761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:20.393791 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:20.396469 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:20.397124 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:20.397129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:20.397132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:20.397255 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:20.397660 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:20.397664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:20.397666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:20.398020 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:20.398178 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:20.398804 (+ 626us) server_negotiation.cc:299] Negotiation successful
0619 13:18:20.398986 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:18:20.400384 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:20.382764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37163 (local address 127.0.97.2:47455)
0619 13:18:20.383442 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:20.383478 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:20.385074 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:20.385741 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:20.385750 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:20.386266 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:20.386804 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:20.386822 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.387819 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.387823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:20.389593 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:20.389609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.390422 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.390430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:20.390661 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:20.391291 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:18:20.391312 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:20.393533 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:20.396603 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:20.396612 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:20.396627 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:20.397007 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:20.397361 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:20.397366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:20.397370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:20.397546 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:20.398148 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:20.398154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:20.398452 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 13:18:20.398683 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 13:18:20.399129 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":265,"threads_started":1}
I20260619 13:18:20.401759 18926 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37163
I20260619 13:18:20.402071 18926 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:20.402814 18926 heartbeater.cc:511] Master 127.0.97.62:37163 requested a full tablet report, sending...
I20260619 13:18:20.404245 18603 ts_manager.cc:194] Registered new tserver with Master: 0c158d085cff4d38bcd2cae52afb9efb (127.0.97.2:36821)
I20260619 13:18:20.405025 18603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47455
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:18:20.440107 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37163
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:20.746889 18931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:20.747164 18931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:20.747298 18931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:20.754442 18931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:20.754541 18931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:20.754696 18931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:18:20.764144 18931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:20.765381 18931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:20.766680 18931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:20.778877 18937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:20.779006 18936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:20.778920 18939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:20.779937 18931 server_base.cc:1034] running on GCE node
I20260619 13:18:20.780511 18931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:20.781407 18931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:20.782596 18931 hybrid_clock.cc:648] HybridClock initialized: now 1781875100782547 us; error 46 us; skew 500 ppm
I20260619 13:18:20.782887 18931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:20 dist-test-slave-mkt3 krb5kdc[18558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:20.785715 18931 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:18:20.787102 18931 webserver.cc:460] Webserver started at http://127.0.97.3:42939/ using document root <none> and password file <none>
I20260619 13:18:20.787743 18931 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:20.787856 18931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:20.788136 18931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:20.791006 18931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "fd3784205aa44854a4ec079f743d51a2"
format_stamp: "Formatted at 2026-06-19 13:18:20 on dist-test-slave-mkt3"
server_key: "252d6ba51d8a97490171527c72562087"
server_key_iv: "d60904d319252181f80e04ce26c5bf42"
server_key_version: "encryptionkey@0"
I20260619 13:18:20.791770 18931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fd3784205aa44854a4ec079f743d51a2"
format_stamp: "Formatted at 2026-06-19 13:18:20 on dist-test-slave-mkt3"
server_key: "252d6ba51d8a97490171527c72562087"
server_key_iv: "d60904d319252181f80e04ce26c5bf42"
server_key_version: "encryptionkey@0"
I20260619 13:18:20.796744 18931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:18:20.800012 18946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:20.801252 18931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:18:20.801386 18931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "fd3784205aa44854a4ec079f743d51a2"
format_stamp: "Formatted at 2026-06-19 13:18:20 on dist-test-slave-mkt3"
server_key: "252d6ba51d8a97490171527c72562087"
server_key_iv: "d60904d319252181f80e04ce26c5bf42"
server_key_version: "encryptionkey@0"
I20260619 13:18:20.801550 18931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:20.827386 18931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:20.831360 18931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:20.831607 18931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:20.832525 18931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:20.833814 18931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:20.833916 18931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:20.834034 18931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:20.834086 18931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:20.863245 18931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45945
I20260619 13:18:20.863279 19058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45945 every 8 connection(s)
I20260619 13:18:20.864432 18931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:18:20.870715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 18931
I20260619 13:18:20.870896 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:18:20.871409 388 external_mini_cluster.cc:1428] Setting key 0f07418f37a0bd632b5b7856587c0aad
Jun 19 13:18:20 dist-test-slave-mkt3 krb5kdc[18558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:18:20.883459 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:20.866814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34809 (local address 127.0.97.62:37163)
0619 13:18:20.866977 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:20.866981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:20.868103 (+ 1122us) server_negotiation.cc:407] Connection header received
0619 13:18:20.869901 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:20.869908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:20.869982 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:20.870069 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:20.871989 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.872941 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.874512 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.874920 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.875028 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:20.878310 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:20.878315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:20.878319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:20.878352 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:20.880555 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:20.881254 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:20.881259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:20.881262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:20.881388 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:20.881785 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:20.881789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:20.881791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:20.882098 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:20.882232 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:20.882907 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 13:18:20.883135 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 13:18:20.884428 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:20.867281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37163 (local address 127.0.97.3:34809)
0619 13:18:20.867931 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:20.867972 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:20.869612 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:20.870310 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:20.870317 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:20.871052 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:20.871812 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:20.871838 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.873088 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.873092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:20.874356 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:20.874373 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.875133 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.875141 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:20.875355 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:20.875919 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:18:20.875941 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:20.878094 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:20.880686 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:20.880695 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:20.880713 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:20.881141 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:20.881490 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:20.881495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:20.881498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:20.881678 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:20.882219 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:20.882228 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:20.882551 (+ 323us) client_negotiation.cc:769] Sending connection context
0619 13:18:20.882797 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 13:18:20.883330 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":256,"threads_started":1}
I20260619 13:18:20.885911 19059 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37163
I20260619 13:18:20.886211 19059 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:20.886914 19059 heartbeater.cc:511] Master 127.0.97.62:37163 requested a full tablet report, sending...
I20260619 13:18:20.888236 18602 ts_manager.cc:194] Registered new tserver with Master: fd3784205aa44854a4ec079f743d51a2 (127.0.97.3:45945)
I20260619 13:18:20.889070 18602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34809
I20260619 13:18:20.898635 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:18:20.915802 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:20.902672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55830 (local address 127.0.97.62:37163)
0619 13:18:20.902819 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:20.902824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:20.903340 (+ 516us) server_negotiation.cc:407] Connection header received
0619 13:18:20.903510 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:20.903514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:20.903594 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:20.903682 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:20.905372 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.906219 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.907468 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.907877 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.907977 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:20.910493 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:20.910498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:20.910502 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:20.910537 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:20.913069 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:20.913800 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:20.913804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:20.913806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:20.913990 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:20.914430 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:20.914433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:20.914435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:20.914744 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:20.914938 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:20.915386 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 13:18:20.915522 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 13:18:20.919869 18602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:18:20.926596 18602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:18:20.946018 18792 heartbeater.cc:503] Master 127.0.97.62:37163 was elected leader, sending a full tablet report...
I20260619 13:18:20.952812 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:20.939517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60788 (local address 127.0.97.2:36821)
0619 13:18:20.940220 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:20.940231 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:20.940879 (+ 648us) server_negotiation.cc:407] Connection header received
0619 13:18:20.940981 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:20.940989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:20.941272 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:20.941460 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:20.942469 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.944148 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.948001 (+ 3853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.950414 (+ 2413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.950914 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:20.951619 (+ 705us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:20.951894 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 13:18:20.952512 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":424,"threads_started":1}
I20260619 13:18:20.953256 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:20.939072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36821 (local address 127.0.0.1:60788)
0619 13:18:20.940632 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:20.940649 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:20.940761 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:20.941737 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:20.941745 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:20.941777 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:20.942256 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:20.942268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.944321 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.944327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:20.947714 (+ 3387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:20.947759 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.950764 (+ 3005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.950797 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:20.951491 (+ 694us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:20.951683 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 13:18:20.952073 (+ 390us) client_negotiation.cc:240] Negotiation successful
0619 13:18:20.952293 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1423,"thread_start_us":1287,"threads_started":1}
I20260619 13:18:20.957464 18861 tablet_service.cc:1467] Processing CreateTablet for tablet 915a9a85375b4377b2c86632b0372f99 (DEFAULT_TABLE table=test-table [id=b66494f12b744aaaa430e15ff3c9ffba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:20.958948 18861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915a9a85375b4377b2c86632b0372f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:20.962155 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:20.940526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45945 (local address 127.0.0.1:39664)
0619 13:18:20.953559 (+ 13033us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:20.953576 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:20.953750 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:20.954703 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:20.954708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:20.954735 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:20.954895 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:20.954905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.957093 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.957098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:20.959371 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:20.959385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.961062 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.961068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:20.961463 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:20.961536 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 13:18:20.961624 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 13:18:20.961769 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12935,"mutex_wait_us":5,"thread_start_us":12775,"threads_started":1}
I20260619 13:18:20.962929 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:20.941013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.0.97.3:45945)
0619 13:18:20.952151 (+ 11138us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:20.952159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:20.953652 (+ 1493us) server_negotiation.cc:407] Connection header received
0619 13:18:20.953966 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:20.953974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:20.954225 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:20.954387 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:20.955066 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.956879 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.959541 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.960908 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.961015 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:20.961250 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:20.962506 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0619 13:18:20.962676 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10642,"thread_start_us":10837,"threads_started":1}
I20260619 13:18:20.963301 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:20.938608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37623 (local address 127.0.0.1:38290)
0619 13:18:20.939378 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:20.939425 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:20.939580 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:20.957106 (+ 17526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:20.957111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:20.957140 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:20.957481 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:20.957493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.958854 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.958859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:20.960688 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:20.960704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.962511 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.962518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:20.962685 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:20.962716 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:18:20.962782 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 13:18:20.962910 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":19,"thread_start_us":1080,"threads_started":1}
I20260619 13:18:20.964283 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:20.938687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.0.97.1:37623)
0619 13:18:20.952598 (+ 13911us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:20.952608 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:20.952645 (+ 37us) server_negotiation.cc:407] Connection header received
0619 13:18:20.952738 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:20.952746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:20.956864 (+ 4118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:20.957236 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:20.957647 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.958726 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.960865 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:20.962302 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:20.963533 (+ 1231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:20.963756 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:20.963831 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 13:18:20.963982 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13742,"thread_start_us":14858,"threads_started":1}
I20260619 13:18:20.966768 18994 tablet_service.cc:1467] Processing CreateTablet for tablet 915a9a85375b4377b2c86632b0372f99 (DEFAULT_TABLE table=test-table [id=b66494f12b744aaaa430e15ff3c9ffba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:20.968288 18994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915a9a85375b4377b2c86632b0372f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:20.967837 18727 tablet_service.cc:1467] Processing CreateTablet for tablet 915a9a85375b4377b2c86632b0372f99 (DEFAULT_TABLE table=test-table [id=b66494f12b744aaaa430e15ff3c9ffba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:20.969103 18727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915a9a85375b4377b2c86632b0372f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:20.971993 19077 tablet_bootstrap.cc:492] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb: Bootstrap starting.
I20260619 13:18:20.974961 19077 tablet_bootstrap.cc:654] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:20.975875 19077 log.cc:824] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:20.978534 19077 tablet_bootstrap.cc:492] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb: No bootstrap required, opened a new log
I20260619 13:18:20.978768 19077 ts_tablet_manager.cc:1397] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 13:18:20.983485 19080 tablet_bootstrap.cc:492] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2: Bootstrap starting.
I20260619 13:18:20.984200 19079 tablet_bootstrap.cc:492] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380: Bootstrap starting.
I20260619 13:18:20.986398 19077 raft_consensus.cc:348] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a20a2848dfb4c94826b0eb5a914e380" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37623 } } peers { permanent_uuid: "fd3784205aa44854a4ec079f743d51a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45945 } } peers { permanent_uuid: "0c158d085cff4d38bcd2cae52afb9efb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36821 } }
I20260619 13:18:20.986814 19077 raft_consensus.cc:374] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:20.986915 19077 raft_consensus.cc:729] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c158d085cff4d38bcd2cae52afb9efb, State: Initialized, Role: FOLLOWER
I20260619 13:18:20.987653 19079 tablet_bootstrap.cc:654] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:20.987506 19077 consensus_queue.cc:260] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a20a2848dfb4c94826b0eb5a914e380" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37623 } } peers { permanent_uuid: "fd3784205aa44854a4ec079f743d51a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45945 } } peers { permanent_uuid: "0c158d085cff4d38bcd2cae52afb9efb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36821 } }
I20260619 13:18:20.988668 19080 tablet_bootstrap.cc:654] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:20.988960 19079 log.cc:824] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:20.989780 19080 log.cc:824] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:20.990386 18926 heartbeater.cc:503] Master 127.0.97.62:37163 was elected leader, sending a full tablet report...
I20260619 13:18:20.991091 19077 ts_tablet_manager.cc:1428] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260619 13:18:20.993259 19079 tablet_bootstrap.cc:492] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380: No bootstrap required, opened a new log
I20260619 13:18:20.993343 19080 tablet_bootstrap.cc:492] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2: No bootstrap required, opened a new log
I20260619 13:18:20.993527 19079 ts_tablet_manager.cc:1397] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 13:18:20.993592 19080 ts_tablet_manager.cc:1397] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260619 13:18:21.001075 19080 raft_consensus.cc:348] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a20a2848dfb4c94826b0eb5a914e380" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37623 } } peers { permanent_uuid: "fd3784205aa44854a4ec079f743d51a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45945 } } peers { permanent_uuid: "0c158d085cff4d38bcd2cae52afb9efb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36821 } }
I20260619 13:18:21.001222 19079 raft_consensus.cc:348] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a20a2848dfb4c94826b0eb5a914e380" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37623 } } peers { permanent_uuid: "fd3784205aa44854a4ec079f743d51a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45945 } } peers { permanent_uuid: "0c158d085cff4d38bcd2cae52afb9efb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36821 } }
I20260619 13:18:21.001543 19080 raft_consensus.cc:374] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:21.001631 19079 raft_consensus.cc:374] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:21.001677 19080 raft_consensus.cc:729] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3784205aa44854a4ec079f743d51a2, State: Initialized, Role: FOLLOWER
I20260619 13:18:21.001734 19079 raft_consensus.cc:729] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a20a2848dfb4c94826b0eb5a914e380, State: Initialized, Role: FOLLOWER
I20260619 13:18:21.002346 19080 consensus_queue.cc:260] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a20a2848dfb4c94826b0eb5a914e380" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37623 } } peers { permanent_uuid: "fd3784205aa44854a4ec079f743d51a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45945 } } peers { permanent_uuid: "0c158d085cff4d38bcd2cae52afb9efb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36821 } }
I20260619 13:18:21.002399 19079 consensus_queue.cc:260] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a20a2848dfb4c94826b0eb5a914e380" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37623 } } peers { permanent_uuid: "fd3784205aa44854a4ec079f743d51a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45945 } } peers { permanent_uuid: "0c158d085cff4d38bcd2cae52afb9efb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36821 } }
I20260619 13:18:21.003473 19059 heartbeater.cc:503] Master 127.0.97.62:37163 was elected leader, sending a full tablet report...
I20260619 13:18:21.004106 19080 ts_tablet_manager.cc:1428] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260619 13:18:21.004108 19079 ts_tablet_manager.cc:1428] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380: Time spent starting tablet: real 0.010s user 0.003s sys 0.009s
I20260619 13:18:21.044721 19084 raft_consensus.cc:482] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:18:21.044991 19084 raft_consensus.cc:504] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a20a2848dfb4c94826b0eb5a914e380" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37623 } } peers { permanent_uuid: "fd3784205aa44854a4ec079f743d51a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45945 } } peers { permanent_uuid: "0c158d085cff4d38bcd2cae52afb9efb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36821 } }
I20260619 13:18:21.046667 19084 leader_election.cc:288] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a20a2848dfb4c94826b0eb5a914e380 (127.0.97.1:37623), 0c158d085cff4d38bcd2cae52afb9efb (127.0.97.2:36821)
I20260619 13:18:21.052626 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:21.047049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46017 (local address 127.0.97.1:37623)
0619 13:18:21.047230 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:21.047235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:21.047254 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:18:21.047528 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:21.047534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:21.047613 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:21.047699 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:21.048109 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.049009 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.050799 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.051887 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.051984 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:21.052056 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:21.052362 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 13:18:21.052449 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:18:21.052784 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:21.046887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37623 (local address 127.0.97.3:46017)
0619 13:18:21.047176 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:21.047197 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:21.047338 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:21.047750 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:21.047753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:21.047779 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:21.047966 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:21.047977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.049138 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.049143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:21.050651 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:21.050663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.052026 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.052032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:21.052214 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:21.052249 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 13:18:21.052347 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 13:18:21.052473 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":31}
I20260619 13:18:21.053544 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:21.047314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54467 (local address 127.0.97.2:36821)
0619 13:18:21.047517 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:21.047522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:21.047849 (+ 327us) server_negotiation.cc:407] Connection header received
0619 13:18:21.047991 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:21.047996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:21.048073 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:21.048168 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:21.048688 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.049549 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.051649 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.052638 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.052760 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:21.052848 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:21.053256 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 13:18:21.053372 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 13:18:21.053651 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:21.046913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36821 (local address 127.0.97.3:54467)
0619 13:18:21.047745 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:21.047763 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:21.047877 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:21.048183 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:21.048187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:21.048210 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:21.048542 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:21.048553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.049687 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.049690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:21.051493 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:21.051509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.052974 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.052980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:21.053108 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:21.053131 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:18:21.053219 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 13:18:21.053303 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":1,"thread_start_us":473,"threads_started":1}
I20260619 13:18:21.053889 18747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915a9a85375b4377b2c86632b0372f99" candidate_uuid: "fd3784205aa44854a4ec079f743d51a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a20a2848dfb4c94826b0eb5a914e380" is_pre_election: true
I20260619 13:18:21.054459 18747 raft_consensus.cc:2454] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3784205aa44854a4ec079f743d51a2 in term 0.
I20260619 13:18:21.054612 18881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915a9a85375b4377b2c86632b0372f99" candidate_uuid: "fd3784205aa44854a4ec079f743d51a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c158d085cff4d38bcd2cae52afb9efb" is_pre_election: true
I20260619 13:18:21.055074 18881 raft_consensus.cc:2454] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3784205aa44854a4ec079f743d51a2 in term 0.
I20260619 13:18:21.055181 18950 leader_election.cc:302] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a20a2848dfb4c94826b0eb5a914e380, fd3784205aa44854a4ec079f743d51a2; no voters:
I20260619 13:18:21.055569 19084 raft_consensus.cc:2781] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:18:21.055730 19084 raft_consensus.cc:482] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:18:21.055855 19084 raft_consensus.cc:3037] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:21.057994 19084 raft_consensus.cc:504] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a20a2848dfb4c94826b0eb5a914e380" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37623 } } peers { permanent_uuid: "fd3784205aa44854a4ec079f743d51a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45945 } } peers { permanent_uuid: "0c158d085cff4d38bcd2cae52afb9efb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36821 } }
I20260619 13:18:21.058534 19084 leader_election.cc:288] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [CANDIDATE]: Term 1 election: Requested vote from peers 1a20a2848dfb4c94826b0eb5a914e380 (127.0.97.1:37623), 0c158d085cff4d38bcd2cae52afb9efb (127.0.97.2:36821)
I20260619 13:18:21.058871 18747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915a9a85375b4377b2c86632b0372f99" candidate_uuid: "fd3784205aa44854a4ec079f743d51a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a20a2848dfb4c94826b0eb5a914e380"
I20260619 13:18:21.058988 18881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915a9a85375b4377b2c86632b0372f99" candidate_uuid: "fd3784205aa44854a4ec079f743d51a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c158d085cff4d38bcd2cae52afb9efb"
I20260619 13:18:21.059039 18747 raft_consensus.cc:3037] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:21.059173 18881 raft_consensus.cc:3037] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:21.061118 18747 raft_consensus.cc:2454] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3784205aa44854a4ec079f743d51a2 in term 1.
I20260619 13:18:21.061395 18881 raft_consensus.cc:2454] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3784205aa44854a4ec079f743d51a2 in term 1.
I20260619 13:18:21.061507 18950 leader_election.cc:302] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a20a2848dfb4c94826b0eb5a914e380, fd3784205aa44854a4ec079f743d51a2; no voters:
I20260619 13:18:21.061782 19084 raft_consensus.cc:2781] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:21.062217 19084 raft_consensus.cc:686] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [term 1 LEADER]: Becoming Leader. State: Replica: fd3784205aa44854a4ec079f743d51a2, State: Running, Role: LEADER
I20260619 13:18:21.062757 19084 consensus_queue.cc:237] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a20a2848dfb4c94826b0eb5a914e380" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37623 } } peers { permanent_uuid: "fd3784205aa44854a4ec079f743d51a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45945 } } peers { permanent_uuid: "0c158d085cff4d38bcd2cae52afb9efb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36821 } }
I20260619 13:18:21.066994 18603 catalog_manager.cc:5495] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd3784205aa44854a4ec079f743d51a2 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fd3784205aa44854a4ec079f743d51a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a20a2848dfb4c94826b0eb5a914e380" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd3784205aa44854a4ec079f743d51a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c158d085cff4d38bcd2cae52afb9efb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36821 } health_report { overall_health: UNKNOWN } } }
I20260619 13:18:21.103569 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:21.098383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.0.97.3:45945)
0619 13:18:21.098675 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:21.098680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:21.098697 (+ 17us) server_negotiation.cc:407] Connection header received
0619 13:18:21.098836 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:21.098840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:21.098915 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:21.099032 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:18:21.099359 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.100182 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.101690 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.102091 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.102179 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:21.102597 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:18:21.103107 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:18:21.103260 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:21.103342 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 13:18:21.103379 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260619 13:18:21.111697 18747 raft_consensus.cc:1261] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380 [term 1 FOLLOWER]: Refusing update from remote peer fd3784205aa44854a4ec079f743d51a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:18:21.111698 18881 raft_consensus.cc:1261] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb [term 1 FOLLOWER]: Refusing update from remote peer fd3784205aa44854a4ec079f743d51a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:18:21.112720 19087 consensus_queue.cc:1035] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c158d085cff4d38bcd2cae52afb9efb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:18:21.113039 19084 consensus_queue.cc:1035] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a20a2848dfb4c94826b0eb5a914e380" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 13:18:21.115788 19060 tablet.cc:2369] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:18:21.126384 19091 mvcc.cc:205] Tried to move back new op lower bound from 7298560414142820352 to 7298560413956718592. Current Snapshot: MvccSnapshot[applied={T|T < 7298560414142820352}]
I20260619 13:18:21.137073 18602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:18:21.137338 18602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:18:21.140588 18602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fc91a310735467383fe049ed33b0514: Sending DeleteTablet for 3 replicas of tablet 915a9a85375b4377b2c86632b0372f99
I20260619 13:18:21.141723 18727 tablet_service.cc:1514] Processing DeleteTablet for tablet 915a9a85375b4377b2c86632b0372f99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38290
I20260619 13:18:21.142041 18861 tablet_service.cc:1514] Processing DeleteTablet for tablet 915a9a85375b4377b2c86632b0372f99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60788
I20260619 13:18:21.141824 18994 tablet_service.cc:1514] Processing DeleteTablet for tablet 915a9a85375b4377b2c86632b0372f99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39664
I20260619 13:18:21.143275 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 18664
I20260619 13:18:21.144783 19101 tablet_replica.cc:330] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2: stopping tablet replica
I20260619 13:18:21.145229 19101 raft_consensus.cc:2229] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:18:21.145943 19101 raft_consensus.cc:2258] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:21.148649 19101 ts_tablet_manager.cc:1905] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:21.153982 19099 tablet_replica.cc:330] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380: stopping tablet replica
I20260619 13:18:21.154407 19099 raft_consensus.cc:2229] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:18:21.154912 19099 raft_consensus.cc:2258] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:21.156672 19101 ts_tablet_manager.cc:1918] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:21.156840 19101 log.cc:1196] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/915a9a85375b4377b2c86632b0372f99
I20260619 13:18:21.156996 19099 ts_tablet_manager.cc:1905] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:21.157351 19101 ts_tablet_manager.cc:1939] T 915a9a85375b4377b2c86632b0372f99 P fd3784205aa44854a4ec079f743d51a2: Deleting consensus metadata
I20260619 13:18:21.159092 18590 catalog_manager.cc:4841] TS fd3784205aa44854a4ec079f743d51a2 (127.0.97.3:45945): tablet 915a9a85375b4377b2c86632b0372f99 (table test-table [id=b66494f12b744aaaa430e15ff3c9ffba]) successfully deleted
I20260619 13:18:21.162319 19099 ts_tablet_manager.cc:1918] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:21.162475 19099 log.cc:1196] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/915a9a85375b4377b2c86632b0372f99
I20260619 13:18:21.162995 19099 ts_tablet_manager.cc:1939] T 915a9a85375b4377b2c86632b0372f99 P 1a20a2848dfb4c94826b0eb5a914e380: Deleting consensus metadata
I20260619 13:18:21.164678 18590 catalog_manager.cc:4841] TS 1a20a2848dfb4c94826b0eb5a914e380 (127.0.97.1:37623): tablet 915a9a85375b4377b2c86632b0372f99 (table test-table [id=b66494f12b744aaaa430e15ff3c9ffba]) successfully deleted
I20260619 13:18:21.167169 19100 tablet_replica.cc:330] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb: stopping tablet replica
I20260619 13:18:21.167629 19100 raft_consensus.cc:2229] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:18:21.168100 19100 raft_consensus.cc:2258] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:21.169766 19100 ts_tablet_manager.cc:1905] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 13:18:21 dist-test-slave-mkt3 krb5kdc[18558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:18:21.175356 19100 ts_tablet_manager.cc:1918] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:21.175498 19100 log.cc:1196] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/915a9a85375b4377b2c86632b0372f99
I20260619 13:18:21.175872 19100 ts_tablet_manager.cc:1939] T 915a9a85375b4377b2c86632b0372f99 P 0c158d085cff4d38bcd2cae52afb9efb: Deleting consensus metadata
I20260619 13:18:21.177505 18588 catalog_manager.cc:4841] TS 0c158d085cff4d38bcd2cae52afb9efb (127.0.97.2:36821): tablet 915a9a85375b4377b2c86632b0372f99 (table test-table [id=b66494f12b744aaaa430e15ff3c9ffba]) successfully deleted
I20260619 13:18:21.179338 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:21.146134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.0.97.1:37623)
0619 13:18:21.146331 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:21.146338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:21.162401 (+ 16063us) server_negotiation.cc:407] Connection header received
0619 13:18:21.162690 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:21.162700 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:21.162784 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:21.162858 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:21.164539 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.165379 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.166735 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.167285 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.167405 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:21.172404 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:21.172420 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:21.172438 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:21.172498 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:21.176295 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:21.177137 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:21.177147 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:21.177151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:21.177305 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:21.177763 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:21.177768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:21.177772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:21.178268 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:21.178562 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:21.178768 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 13:18:21.178986 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:18:21.180006 18787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:21.268846 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 18664
I20260619 13:18:21.282967 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 18798
Jun 19 13:18:21 dist-test-slave-mkt3 krb5kdc[18558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:18:21.297641 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:21.283812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.0.97.2:36821)
0619 13:18:21.283997 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:21.284003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:21.284039 (+ 36us) server_negotiation.cc:407] Connection header received
0619 13:18:21.284099 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:21.284103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:21.284177 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:21.284259 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:21.285447 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.286232 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.287493 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.287878 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.287987 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:21.292319 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:21.292334 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:21.292349 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:21.292384 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:21.295096 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:21.295862 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:21.295870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:21.295872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:21.296008 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:21.296355 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:21.296358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:21.296360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:21.296718 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:21.296923 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:21.297224 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 13:18:21.297376 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:18:21.298597 18921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:21.395218 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 18798
I20260619 13:18:21.409387 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 18931
Jun 19 13:18:21 dist-test-slave-mkt3 krb5kdc[18558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:18:21.426730 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:21.410231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.0.97.3:45945)
0619 13:18:21.410395 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:21.410400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:21.410416 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:18:21.410576 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:21.410579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:21.410639 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:21.410723 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:21.412463 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.413278 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.415220 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:21.415639 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:21.415736 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:21.421045 (+ 5309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:21.421059 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:21.421075 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:21.421113 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:21.424083 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:21.424852 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:21.424859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:21.424861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:21.425014 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:21.425385 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:21.425388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:21.425391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:21.425744 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:21.425998 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:21.426293 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 13:18:21.426450 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:18:21.427619 19054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:21.552141 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 18931
I20260619 13:18:21.567656 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 18571
I20260619 13:18:21.568714 18633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:21.676713 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 18571
2026-06-19T13:18:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (3996 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T13:18:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:21Z Disabled control of system clock
I20260619 13:18:21.743228 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45893
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:18:22.054818 19110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:22.055099 19110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:22.055217 19110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:22.062381 19110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:22.062490 19110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:22.062549 19110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:22.062623 19110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:22.062681 19110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:22.071869 19110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:22.073117 19110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:22.074927 19110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:22.086012 19118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:22.086045 19116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:22.086012 19115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:22.086810 19110 server_base.cc:1034] running on GCE node
I20260619 13:18:22.087582 19110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:22.088965 19110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:22.090173 19110 hybrid_clock.cc:648] HybridClock initialized: now 1781875102090124 us; error 43 us; skew 500 ppm
I20260619 13:18:22.090493 19110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:18:22.092641 19110 webserver.cc:460] Webserver started at http://127.0.97.62:33685/ using document root <none> and password file <none>
I20260619 13:18:22.093299 19110 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:22.093397 19110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:22.093688 19110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:22.096556 19110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "f9459915fbbf4a6183324f3e2d14841b"
format_stamp: "Formatted at 2026-06-19 13:18:22 on dist-test-slave-mkt3"
server_key: "1a0fe861c72d12acf5fac0c3ceb7f874"
server_key_iv: "c888c869ce20fd0251b234a2128934a7"
server_key_version: "encryptionkey@0"
I20260619 13:18:22.097302 19110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "f9459915fbbf4a6183324f3e2d14841b"
format_stamp: "Formatted at 2026-06-19 13:18:22 on dist-test-slave-mkt3"
server_key: "1a0fe861c72d12acf5fac0c3ceb7f874"
server_key_iv: "c888c869ce20fd0251b234a2128934a7"
server_key_version: "encryptionkey@0"
I20260619 13:18:22.102227 19110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:18:22.105589 19124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:22.106874 19110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:18:22.107029 19110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "f9459915fbbf4a6183324f3e2d14841b"
format_stamp: "Formatted at 2026-06-19 13:18:22 on dist-test-slave-mkt3"
server_key: "1a0fe861c72d12acf5fac0c3ceb7f874"
server_key_iv: "c888c869ce20fd0251b234a2128934a7"
server_key_version: "encryptionkey@0"
I20260619 13:18:22.107234 19110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:22.119184 19110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:22.119908 19110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:22.120157 19110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:22.136518 19110 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45893
I20260619 13:18:22.136539 19175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45893 every 8 connection(s)
I20260619 13:18:22.137738 19110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:22.141289 19176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:22.145146 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 19110
I20260619 13:18:22.145299 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:22.145714 388 external_mini_cluster.cc:1428] Setting key 3025c24bed073886dfd0eae9e49dd25e
I20260619 13:18:22.151909 19176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b: Bootstrap starting.
I20260619 13:18:22.155387 19176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:22.156569 19176 log.cc:824] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:22.159634 19179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:22.147488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.0.97.62:45893)
0619 13:18:22.148388 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:22.148401 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:22.148445 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:18:22.150430 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:22.150451 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:22.150889 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:22.151210 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:22.151791 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:22.153098 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:22.155039 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:22.155804 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:22.155932 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:22.157172 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:22.157185 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:22.157205 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:22.157978 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:22.158071 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:22.158091 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:22.158178 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:22.158400 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 13:18:22.158702 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":293,"threads_started":1}
I20260619 13:18:22.159680 19176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b: No bootstrap required, opened a new log
I20260619 13:18:22.166716 19176 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9459915fbbf4a6183324f3e2d14841b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45893 } }
I20260619 13:18:22.167071 19176 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:22.167184 19176 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9459915fbbf4a6183324f3e2d14841b, State: Initialized, Role: FOLLOWER
I20260619 13:18:22.167754 19176 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9459915fbbf4a6183324f3e2d14841b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45893 } }
I20260619 13:18:22.167966 19176 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:22.168058 19176 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:22.168180 19176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:22.170293 19176 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9459915fbbf4a6183324f3e2d14841b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45893 } }
I20260619 13:18:22.170782 19176 leader_election.cc:302] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9459915fbbf4a6183324f3e2d14841b; no voters:
I20260619 13:18:22.171283 19176 leader_election.cc:288] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:22.171407 19181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:22.172020 19181 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [term 1 LEADER]: Becoming Leader. State: Replica: f9459915fbbf4a6183324f3e2d14841b, State: Running, Role: LEADER
I20260619 13:18:22.172489 19181 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9459915fbbf4a6183324f3e2d14841b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45893 } }
I20260619 13:18:22.173252 19176 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:22.175092 19183 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9459915fbbf4a6183324f3e2d14841b. Latest consensus state: current_term: 1 leader_uuid: "f9459915fbbf4a6183324f3e2d14841b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9459915fbbf4a6183324f3e2d14841b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45893 } } }
I20260619 13:18:22.175454 19183 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:22.177338 19182 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9459915fbbf4a6183324f3e2d14841b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9459915fbbf4a6183324f3e2d14841b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45893 } } }
I20260619 13:18:22.177553 19182 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:22.177718 19187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:22.183369 19187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:22.193341 19187 catalog_manager.cc:1347] Generated new cluster ID: 76ef60d7ddfe4d8e99c9e8fae872afb9
I20260619 13:18:22.193468 19187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:22.207378 19187 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:22.209061 19187 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:22.225636 19187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b: Generated new TSK 0
I20260619 13:18:22.226679 19187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:22.240607 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45893
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:18:22.551133 19200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:22.551432 19200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:22.551558 19200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:22.558645 19200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:22.558748 19200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:22.558892 19200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:18:22.567735 19200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:22.568956 19200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:22.570667 19200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:22.582414 19205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:22.582463 19208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:22.582468 19206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:22.583516 19200 server_base.cc:1034] running on GCE node
I20260619 13:18:22.584026 19200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:22.584817 19200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:22.585997 19200 hybrid_clock.cc:648] HybridClock initialized: now 1781875102585945 us; error 42 us; skew 500 ppm
I20260619 13:18:22.586283 19200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:18:22.588581 19200 webserver.cc:460] Webserver started at http://127.0.97.1:36131/ using document root <none> and password file <none>
I20260619 13:18:22.589187 19200 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:22.589285 19200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:22.589569 19200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:22.592264 19200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "f0c11a59edbb4d7eb8da90fa20806a45"
format_stamp: "Formatted at 2026-06-19 13:18:22 on dist-test-slave-mkt3"
server_key: "3dc4652cc485020503d9b04a2ef9f756"
server_key_iv: "e19d6d5e4d7731b7e5c94cb3c88fbaac"
server_key_version: "encryptionkey@0"
I20260619 13:18:22.592986 19200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f0c11a59edbb4d7eb8da90fa20806a45"
format_stamp: "Formatted at 2026-06-19 13:18:22 on dist-test-slave-mkt3"
server_key: "3dc4652cc485020503d9b04a2ef9f756"
server_key_iv: "e19d6d5e4d7731b7e5c94cb3c88fbaac"
server_key_version: "encryptionkey@0"
I20260619 13:18:22.597671 19200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 13:18:22.600852 19214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:22.602133 19200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:18:22.602288 19200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "f0c11a59edbb4d7eb8da90fa20806a45"
format_stamp: "Formatted at 2026-06-19 13:18:22 on dist-test-slave-mkt3"
server_key: "3dc4652cc485020503d9b04a2ef9f756"
server_key_iv: "e19d6d5e4d7731b7e5c94cb3c88fbaac"
server_key_version: "encryptionkey@0"
I20260619 13:18:22.602473 19200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:22.627266 19200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:22.628013 19200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:22.628429 19200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:22.629392 19200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:22.630824 19200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:22.630918 19200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:22.631026 19200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:22.631105 19200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:22.660475 19200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44299
I20260619 13:18:22.660534 19326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44299 every 8 connection(s)
I20260619 13:18:22.661796 19200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:18:22.664541 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 19200
I20260619 13:18:22.664713 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:18:22.665234 388 external_mini_cluster.cc:1428] Setting key 17ee4f06eeaf282f29f39a6004d3dd7c
I20260619 13:18:22.668059 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45893
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:18:22.675122 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:22.664284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37505 (local address 127.0.97.62:45893)
0619 13:18:22.664718 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:22.664723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:22.666191 (+ 1468us) server_negotiation.cc:407] Connection header received
0619 13:18:22.668319 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:22.668325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:22.668402 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:22.668493 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:22.669760 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:22.670572 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:22.672249 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:22.672773 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:22.672880 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:22.674275 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:22.674280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:22.674284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:22.674301 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:22.674341 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:22.674352 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:22.674445 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:22.674686 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 13:18:22.674780 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":214,"threads_started":1}
I20260619 13:18:22.676195 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:22.664960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45893 (local address 127.0.97.1:37505)
0619 13:18:22.665987 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:22.666037 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:22.668007 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:22.668741 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:22.668749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:22.668848 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:22.669580 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:22.669602 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:22.670715 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:22.670721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:22.672128 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:22.672143 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:22.672985 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:22.672992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:22.673249 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:22.673947 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:18:22.673950 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:22.673988 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:18:22.674076 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:22.674488 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:22.674498 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:22.674504 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 13:18:22.674659 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 13:18:22.675042 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":307,"threads_started":1}
I20260619 13:18:22.677732 19327 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45893
I20260619 13:18:22.678035 19327 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:22.678771 19327 heartbeater.cc:511] Master 127.0.97.62:45893 requested a full tablet report, sending...
I20260619 13:18:22.680998 19141 ts_manager.cc:194] Registered new tserver with Master: f0c11a59edbb4d7eb8da90fa20806a45 (127.0.97.1:44299)
I20260619 13:18:22.682940 19141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:37505
W20260619 13:18:22.976248 19331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:22.976500 19331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:22.976580 19331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:22.983603 19331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:22.983716 19331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:22.983855 19331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:18:22.992679 19331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:22.993754 19331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:22.995409 19331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:23.007658 19336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:23.007553 19339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:23.007661 19337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:23.008895 19331 server_base.cc:1034] running on GCE node
I20260619 13:18:23.009511 19331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:23.010490 19331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:23.011684 19331 hybrid_clock.cc:648] HybridClock initialized: now 1781875103011623 us; error 54 us; skew 500 ppm
I20260619 13:18:23.011976 19331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:18:23.014150 19331 webserver.cc:460] Webserver started at http://127.0.97.2:42725/ using document root <none> and password file <none>
I20260619 13:18:23.014808 19331 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:23.014905 19331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:23.015208 19331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:23.017979 19331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "232ce9e786c9454caabcb0bd2a4810cd"
format_stamp: "Formatted at 2026-06-19 13:18:23 on dist-test-slave-mkt3"
server_key: "97493d3bfc4274a45d4c1dbc24919384"
server_key_iv: "5640b50a7af6e7c63588820c04e27f15"
server_key_version: "encryptionkey@0"
I20260619 13:18:23.018733 19331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "232ce9e786c9454caabcb0bd2a4810cd"
format_stamp: "Formatted at 2026-06-19 13:18:23 on dist-test-slave-mkt3"
server_key: "97493d3bfc4274a45d4c1dbc24919384"
server_key_iv: "5640b50a7af6e7c63588820c04e27f15"
server_key_version: "encryptionkey@0"
I20260619 13:18:23.023515 19331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:18:23.026906 19345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:23.028245 19331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:18:23.028393 19331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "232ce9e786c9454caabcb0bd2a4810cd"
format_stamp: "Formatted at 2026-06-19 13:18:23 on dist-test-slave-mkt3"
server_key: "97493d3bfc4274a45d4c1dbc24919384"
server_key_iv: "5640b50a7af6e7c63588820c04e27f15"
server_key_version: "encryptionkey@0"
I20260619 13:18:23.028579 19331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:23.055042 19331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:23.055845 19331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:23.056084 19331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:23.057000 19331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:23.058458 19331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:23.058550 19331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:23.058651 19331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:23.058722 19331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:23.087944 19331 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35923
I20260619 13:18:23.088007 19457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35923 every 8 connection(s)
I20260619 13:18:23.089232 19331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:18:23.091917 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 19331
I20260619 13:18:23.092051 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:18:23.092456 388 external_mini_cluster.cc:1428] Setting key bd631711d6685e8e776637960ebbb9ae
I20260619 13:18:23.095422 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45893
--builtin_ntp_servers=127.0.97.20:34235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:18:23.102100 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.091605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37433 (local address 127.0.97.62:45893)
0619 13:18:23.091827 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:23.091833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:23.093376 (+ 1543us) server_negotiation.cc:407] Connection header received
0619 13:18:23.095664 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:23.095669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:23.095749 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:23.095837 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:23.097168 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.097929 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.099497 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.099899 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.100009 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.101306 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:23.101312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:23.101316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:23.101332 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:23.101363 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:23.101374 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:23.101444 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:23.101656 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 13:18:23.101753 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 13:18:23.103089 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.092193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45893 (local address 127.0.97.2:37433)
0619 13:18:23.093197 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:23.093253 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:23.095343 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:23.096085 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:23.096096 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:23.096225 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:23.096994 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:23.097016 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.098086 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.098092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:23.099357 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:23.099369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.100138 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.100147 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:23.100355 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.100993 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:18:23.100995 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:23.101048 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:18:23.101129 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:23.101456 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:23.101467 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:23.101475 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 13:18:23.101632 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 13:18:23.102061 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":268,"threads_started":1}
I20260619 13:18:23.104437 19458 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45893
I20260619 13:18:23.104707 19458 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:23.105427 19458 heartbeater.cc:511] Master 127.0.97.62:45893 requested a full tablet report, sending...
I20260619 13:18:23.106858 19141 ts_manager.cc:194] Registered new tserver with Master: 232ce9e786c9454caabcb0bd2a4810cd (127.0.97.2:35923)
I20260619 13:18:23.107851 19141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:37433
W20260619 13:18:23.406186 19461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:23.406481 19461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:23.406612 19461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:23.413757 19461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:23.413867 19461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:23.414017 19461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:18:23.422783 19461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:23.423952 19461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:23.425621 19461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:23.438185 19467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:23.438164 19466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:23.438124 19469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:23.438922 19461 server_base.cc:1034] running on GCE node
I20260619 13:18:23.439527 19461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:23.440434 19461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:23.441659 19461 hybrid_clock.cc:648] HybridClock initialized: now 1781875103441601 us; error 49 us; skew 500 ppm
I20260619 13:18:23.442027 19461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:18:23.444212 19461 webserver.cc:460] Webserver started at http://127.0.97.3:32789/ using document root <none> and password file <none>
I20260619 13:18:23.444891 19461 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:23.444993 19461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:23.445298 19461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:23.448163 19461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "28b576eb79094049811af6b44ebb2ec5"
format_stamp: "Formatted at 2026-06-19 13:18:23 on dist-test-slave-mkt3"
server_key: "fa163b9f1cddbea6698eb57c32065028"
server_key_iv: "167591d81ebe1201aad95f5f0db2c710"
server_key_version: "encryptionkey@0"
I20260619 13:18:23.449177 19461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "28b576eb79094049811af6b44ebb2ec5"
format_stamp: "Formatted at 2026-06-19 13:18:23 on dist-test-slave-mkt3"
server_key: "fa163b9f1cddbea6698eb57c32065028"
server_key_iv: "167591d81ebe1201aad95f5f0db2c710"
server_key_version: "encryptionkey@0"
I20260619 13:18:23.454406 19461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 13:18:23.457746 19475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:23.459064 19461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:18:23.459229 19461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "28b576eb79094049811af6b44ebb2ec5"
format_stamp: "Formatted at 2026-06-19 13:18:23 on dist-test-slave-mkt3"
server_key: "fa163b9f1cddbea6698eb57c32065028"
server_key_iv: "167591d81ebe1201aad95f5f0db2c710"
server_key_version: "encryptionkey@0"
I20260619 13:18:23.459426 19461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:23.477586 19461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:23.478417 19461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:23.478664 19461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:23.479863 19461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:23.481319 19461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:23.481403 19461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:23.481500 19461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:23.481561 19461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:23.510462 19461 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39757
I20260619 13:18:23.510543 19587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39757 every 8 connection(s)
I20260619 13:18:23.511782 19461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:18:23.518543 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 19461
I20260619 13:18:23.518723 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:18:23.519127 388 external_mini_cluster.cc:1428] Setting key d03c11b536f7948c43a49f56182c7a02
I20260619 13:18:23.523975 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.514175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46133 (local address 127.0.97.62:45893)
0619 13:18:23.514370 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:23.514374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:23.515723 (+ 1349us) server_negotiation.cc:407] Connection header received
0619 13:18:23.517378 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:23.517382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:23.517444 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:23.517526 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:23.518726 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.519514 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.521148 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.521782 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.521933 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.523218 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:23.523222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:23.523226 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:23.523244 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:23.523292 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:23.523304 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:23.523380 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:23.523636 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 13:18:23.523719 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:18:23.524883 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.514731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45893 (local address 127.0.97.3:46133)
0619 13:18:23.515584 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:23.515619 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:23.517163 (+ 1544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:23.517718 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:23.517727 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:23.517808 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:23.518543 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:23.518571 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.519662 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.519666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:23.521001 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:23.521014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.522007 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.522012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:23.522224 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.522883 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:18:23.522893 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:23.522933 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:18:23.523026 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:23.523410 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:23.523419 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:23.523426 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 13:18:23.523582 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 13:18:23.523876 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":255,"threads_started":1}
I20260619 13:18:23.526149 19588 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45893
I20260619 13:18:23.526405 19588 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:23.527104 19588 heartbeater.cc:511] Master 127.0.97.62:45893 requested a full tablet report, sending...
I20260619 13:18:23.528378 19141 ts_manager.cc:194] Registered new tserver with Master: 28b576eb79094049811af6b44ebb2ec5 (127.0.97.3:39757)
I20260619 13:18:23.529112 19141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:46133
I20260619 13:18:23.533766 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:18:23.542909 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.537472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.0.97.62:45893)
0619 13:18:23.537683 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:23.537689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:23.538006 (+ 317us) server_negotiation.cc:407] Connection header received
0619 13:18:23.538135 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:23.538139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:23.538195 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:23.538271 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:23.538673 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.539358 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.540709 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.541088 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.541172 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.542183 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:23.542188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:23.542191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:23.542225 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:23.542254 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:23.542266 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:23.542379 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:23.542577 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 13:18:23.542620 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 13:18:23.546274 19141 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:18:23.552759 19141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:18:23.586771 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.564197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35923 (local address 127.0.0.1:56236)
0619 13:18:23.575720 (+ 11523us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:23.575781 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:23.575979 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:23.577987 (+ 2008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:23.577994 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:23.578028 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:23.578494 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:23.578508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.581014 (+ 2506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.581034 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:23.583652 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:23.583669 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.585684 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.585691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:23.585925 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.585986 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 13:18:23.586102 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 13:18:23.586315 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11208,"mutex_wait_us":1,"thread_start_us":13507,"threads_started":1}
I20260619 13:18:23.587690 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.564863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44299 (local address 127.0.0.1:39942)
0619 13:18:23.577954 (+ 13091us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:23.577968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:23.578124 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:23.579435 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:23.579441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:23.579469 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:23.579893 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:23.579905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.582033 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.582039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:23.584367 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:23.584382 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.586773 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.586780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:23.587005 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.587040 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 13:18:23.587127 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 13:18:23.587267 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12941,"spinlock_wait_cycles":7424,"thread_start_us":11321,"threads_started":1}
I20260619 13:18:23.588788 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.565017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.0.97.1:44299)
0619 13:18:23.577576 (+ 12559us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:23.577582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:23.578078 (+ 496us) server_negotiation.cc:407] Connection header received
0619 13:18:23.578314 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:23.578322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:23.578552 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:23.578755 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:23.580097 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.581816 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.584600 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.586175 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.586285 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.586554 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:23.588186 (+ 1632us) server_negotiation.cc:299] Negotiation successful
0619 13:18:23.588398 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12413,"thread_start_us":12298,"threads_started":1}
I20260619 13:18:23.588786 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.564540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.0.97.2:35923)
0619 13:18:23.577268 (+ 12728us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:23.577278 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:23.577315 (+ 37us) server_negotiation.cc:407] Connection header received
0619 13:18:23.577424 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:23.577432 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:23.577721 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:23.578726 (+ 1005us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:23.579050 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.580855 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.583901 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.585435 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.587660 (+ 2225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.588007 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:23.588082 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 13:18:23.588277 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12540,"thread_start_us":12413,"threads_started":1}
I20260619 13:18:23.593031 19262 tablet_service.cc:1467] Processing CreateTablet for tablet 9126a9efa5ec490abc3215d5dc07956c (DEFAULT_TABLE table=test-table [id=9f1992fc528b483a906370c96abefd98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:23.593037 19393 tablet_service.cc:1467] Processing CreateTablet for tablet 9126a9efa5ec490abc3215d5dc07956c (DEFAULT_TABLE table=test-table [id=9f1992fc528b483a906370c96abefd98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:23.594375 19393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9126a9efa5ec490abc3215d5dc07956c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:23.594408 19262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9126a9efa5ec490abc3215d5dc07956c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:23.598814 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.578379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39757 (local address 127.0.0.1:52590)
0619 13:18:23.587323 (+ 8944us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:23.587343 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:23.587450 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:23.590600 (+ 3150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:23.590604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:23.590629 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:23.590789 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:23.590800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.594600 (+ 3800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.594606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:23.596344 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:23.596359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.598074 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.598080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:23.598267 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.598303 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 13:18:23.598404 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 13:18:23.598540 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8830,"thread_start_us":10798,"threads_started":1}
I20260619 13:18:23.598955 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.578494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52590 (local address 127.0.97.3:39757)
0619 13:18:23.590020 (+ 11526us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:23.590028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:23.590064 (+ 36us) server_negotiation.cc:407] Connection header received
0619 13:18:23.590166 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:23.590173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:23.590422 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:23.590602 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:23.592832 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.594416 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.596533 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.597915 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.598060 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.598323 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:23.598439 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 13:18:23.598604 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11371,"thread_start_us":11211,"threads_started":1}
I20260619 13:18:23.602862 19523 tablet_service.cc:1467] Processing CreateTablet for tablet 9126a9efa5ec490abc3215d5dc07956c (DEFAULT_TABLE table=test-table [id=9f1992fc528b483a906370c96abefd98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:23.604348 19523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9126a9efa5ec490abc3215d5dc07956c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:23.607146 19606 tablet_bootstrap.cc:492] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45: Bootstrap starting.
I20260619 13:18:23.610415 19606 tablet_bootstrap.cc:654] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:23.611584 19606 log.cc:824] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:23.618973 19607 tablet_bootstrap.cc:492] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd: Bootstrap starting.
I20260619 13:18:23.622604 19607 tablet_bootstrap.cc:654] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:23.623819 19607 log.cc:824] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:23.624629 19606 tablet_bootstrap.cc:492] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45: No bootstrap required, opened a new log
I20260619 13:18:23.624619 19609 tablet_bootstrap.cc:492] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5: Bootstrap starting.
I20260619 13:18:23.624894 19606 ts_tablet_manager.cc:1397] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.012s
I20260619 13:18:23.627813 19609 tablet_bootstrap.cc:654] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:23.628965 19609 log.cc:824] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:23.629243 19607 tablet_bootstrap.cc:492] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd: No bootstrap required, opened a new log
I20260619 13:18:23.629487 19607 ts_tablet_manager.cc:1397] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260619 13:18:23.631865 19609 tablet_bootstrap.cc:492] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5: No bootstrap required, opened a new log
I20260619 13:18:23.632099 19609 ts_tablet_manager.cc:1397] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260619 13:18:23.634416 19606 raft_consensus.cc:348] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232ce9e786c9454caabcb0bd2a4810cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35923 } } peers { permanent_uuid: "28b576eb79094049811af6b44ebb2ec5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39757 } } peers { permanent_uuid: "f0c11a59edbb4d7eb8da90fa20806a45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260619 13:18:23.634894 19606 raft_consensus.cc:374] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:23.634999 19606 raft_consensus.cc:729] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0c11a59edbb4d7eb8da90fa20806a45, State: Initialized, Role: FOLLOWER
I20260619 13:18:23.635600 19606 consensus_queue.cc:260] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232ce9e786c9454caabcb0bd2a4810cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35923 } } peers { permanent_uuid: "28b576eb79094049811af6b44ebb2ec5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39757 } } peers { permanent_uuid: "f0c11a59edbb4d7eb8da90fa20806a45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260619 13:18:23.636760 19327 heartbeater.cc:503] Master 127.0.97.62:45893 was elected leader, sending a full tablet report...
I20260619 13:18:23.637385 19606 ts_tablet_manager.cc:1428] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260619 13:18:23.637632 19607 raft_consensus.cc:348] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232ce9e786c9454caabcb0bd2a4810cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35923 } } peers { permanent_uuid: "28b576eb79094049811af6b44ebb2ec5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39757 } } peers { permanent_uuid: "f0c11a59edbb4d7eb8da90fa20806a45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260619 13:18:23.638055 19607 raft_consensus.cc:374] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:23.638166 19607 raft_consensus.cc:729] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 232ce9e786c9454caabcb0bd2a4810cd, State: Initialized, Role: FOLLOWER
I20260619 13:18:23.638749 19607 consensus_queue.cc:260] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232ce9e786c9454caabcb0bd2a4810cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35923 } } peers { permanent_uuid: "28b576eb79094049811af6b44ebb2ec5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39757 } } peers { permanent_uuid: "f0c11a59edbb4d7eb8da90fa20806a45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260619 13:18:23.641232 19458 heartbeater.cc:503] Master 127.0.97.62:45893 was elected leader, sending a full tablet report...
I20260619 13:18:23.641860 19607 ts_tablet_manager.cc:1428] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd: Time spent starting tablet: real 0.012s user 0.001s sys 0.010s
I20260619 13:18:23.641589 19609 raft_consensus.cc:348] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232ce9e786c9454caabcb0bd2a4810cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35923 } } peers { permanent_uuid: "28b576eb79094049811af6b44ebb2ec5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39757 } } peers { permanent_uuid: "f0c11a59edbb4d7eb8da90fa20806a45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260619 13:18:23.642015 19609 raft_consensus.cc:374] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:23.642105 19609 raft_consensus.cc:729] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b576eb79094049811af6b44ebb2ec5, State: Initialized, Role: FOLLOWER
I20260619 13:18:23.642673 19609 consensus_queue.cc:260] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232ce9e786c9454caabcb0bd2a4810cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35923 } } peers { permanent_uuid: "28b576eb79094049811af6b44ebb2ec5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39757 } } peers { permanent_uuid: "f0c11a59edbb4d7eb8da90fa20806a45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260619 13:18:23.643673 19588 heartbeater.cc:503] Master 127.0.97.62:45893 was elected leader, sending a full tablet report...
I20260619 13:18:23.644263 19609 ts_tablet_manager.cc:1428] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260619 13:18:23.663832 19328 tablet.cc:2369] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:18:23.763358 19589 tablet.cc:2369] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:18:23.838490 19614 raft_consensus.cc:482] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:18:23.838770 19614 raft_consensus.cc:504] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232ce9e786c9454caabcb0bd2a4810cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35923 } } peers { permanent_uuid: "28b576eb79094049811af6b44ebb2ec5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39757 } } peers { permanent_uuid: "f0c11a59edbb4d7eb8da90fa20806a45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260619 13:18:23.840404 19614 leader_election.cc:288] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 232ce9e786c9454caabcb0bd2a4810cd (127.0.97.2:35923), f0c11a59edbb4d7eb8da90fa20806a45 (127.0.97.1:44299)
W20260619 13:18:23.841348 19459 tablet.cc:2369] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:18:23.846078 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.840784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57881 (local address 127.0.97.2:35923)
0619 13:18:23.840989 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:23.840994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:23.841030 (+ 36us) server_negotiation.cc:407] Connection header received
0619 13:18:23.841241 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:23.841247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:23.841334 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:23.841418 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:23.842032 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.842933 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.844535 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.845258 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.845366 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.845470 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:23.845720 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 13:18:23.845835 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:18:23.846146 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.840632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35923 (local address 127.0.97.3:57881)
0619 13:18:23.840835 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:23.840855 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:23.841040 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:23.841613 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:23.841617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:23.841644 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:23.841896 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:23.841907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.843047 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.843053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:23.844388 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:23.844403 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.845397 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.845403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:23.845576 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.845599 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 13:18:23.845673 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 13:18:23.845809 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":1,"spinlock_wait_cycles":15616}
I20260619 13:18:23.846465 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.840827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52751 (local address 127.0.97.1:44299)
0619 13:18:23.841010 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:23.841030 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:23.841646 (+ 616us) server_negotiation.cc:407] Connection header received
0619 13:18:23.841905 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:23.841910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:23.841998 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:23.842102 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:23.842594 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.843246 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.844931 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.845661 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.845768 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.845873 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:23.846121 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 13:18:23.846243 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 13:18:23.846554 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.840638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44299 (local address 127.0.97.3:52751)
0619 13:18:23.841539 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:23.841554 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:23.841742 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:23.842116 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:23.842121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:23.842144 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:23.842462 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:23.842473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.843391 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.843396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:23.844766 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:23.844778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.845813 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.845819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:23.845990 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.846012 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 13:18:23.846084 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 13:18:23.846212 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":256,"spinlock_wait_cycles":48000,"thread_start_us":312,"threads_started":1}
I20260619 13:18:23.847085 19413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9126a9efa5ec490abc3215d5dc07956c" candidate_uuid: "28b576eb79094049811af6b44ebb2ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "232ce9e786c9454caabcb0bd2a4810cd" is_pre_election: true
I20260619 13:18:23.847453 19282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9126a9efa5ec490abc3215d5dc07956c" candidate_uuid: "28b576eb79094049811af6b44ebb2ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0c11a59edbb4d7eb8da90fa20806a45" is_pre_election: true
I20260619 13:18:23.847666 19413 raft_consensus.cc:2454] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b576eb79094049811af6b44ebb2ec5 in term 0.
I20260619 13:18:23.847968 19282 raft_consensus.cc:2454] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28b576eb79094049811af6b44ebb2ec5 in term 0.
I20260619 13:18:23.848358 19476 leader_election.cc:302] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 232ce9e786c9454caabcb0bd2a4810cd, 28b576eb79094049811af6b44ebb2ec5; no voters:
I20260619 13:18:23.848742 19614 raft_consensus.cc:2781] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:18:23.848881 19614 raft_consensus.cc:482] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:18:23.848979 19614 raft_consensus.cc:3037] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:23.851084 19614 raft_consensus.cc:504] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232ce9e786c9454caabcb0bd2a4810cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35923 } } peers { permanent_uuid: "28b576eb79094049811af6b44ebb2ec5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39757 } } peers { permanent_uuid: "f0c11a59edbb4d7eb8da90fa20806a45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260619 13:18:23.851658 19614 leader_election.cc:288] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [CANDIDATE]: Term 1 election: Requested vote from peers 232ce9e786c9454caabcb0bd2a4810cd (127.0.97.2:35923), f0c11a59edbb4d7eb8da90fa20806a45 (127.0.97.1:44299)
I20260619 13:18:23.852049 19413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9126a9efa5ec490abc3215d5dc07956c" candidate_uuid: "28b576eb79094049811af6b44ebb2ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "232ce9e786c9454caabcb0bd2a4810cd"
I20260619 13:18:23.852147 19282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9126a9efa5ec490abc3215d5dc07956c" candidate_uuid: "28b576eb79094049811af6b44ebb2ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0c11a59edbb4d7eb8da90fa20806a45"
I20260619 13:18:23.852210 19413 raft_consensus.cc:3037] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:23.852339 19282 raft_consensus.cc:3037] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:23.854363 19413 raft_consensus.cc:2454] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b576eb79094049811af6b44ebb2ec5 in term 1.
I20260619 13:18:23.854467 19282 raft_consensus.cc:2454] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28b576eb79094049811af6b44ebb2ec5 in term 1.
I20260619 13:18:23.854735 19476 leader_election.cc:302] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 232ce9e786c9454caabcb0bd2a4810cd, 28b576eb79094049811af6b44ebb2ec5; no voters:
I20260619 13:18:23.855002 19614 raft_consensus.cc:2781] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:23.855406 19614 raft_consensus.cc:686] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [term 1 LEADER]: Becoming Leader. State: Replica: 28b576eb79094049811af6b44ebb2ec5, State: Running, Role: LEADER
I20260619 13:18:23.855862 19614 consensus_queue.cc:237] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232ce9e786c9454caabcb0bd2a4810cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35923 } } peers { permanent_uuid: "28b576eb79094049811af6b44ebb2ec5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39757 } } peers { permanent_uuid: "f0c11a59edbb4d7eb8da90fa20806a45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } }
I20260619 13:18:23.860550 19141 catalog_manager.cc:5495] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28b576eb79094049811af6b44ebb2ec5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "28b576eb79094049811af6b44ebb2ec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232ce9e786c9454caabcb0bd2a4810cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28b576eb79094049811af6b44ebb2ec5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0c11a59edbb4d7eb8da90fa20806a45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 } health_report { overall_health: UNKNOWN } } }
I20260619 13:18:23.893373 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.888515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52592 (local address 127.0.97.3:39757)
0619 13:18:23.888692 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:23.888697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:23.888713 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:18:23.888789 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:23.888793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:23.888866 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:23.888983 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:18:23.889328 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.890088 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.891418 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.891848 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.891950 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.892377 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:18:23.892879 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:18:23.893011 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:23.893088 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 13:18:23.893128 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:18:23.901209 19413 raft_consensus.cc:1261] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd [term 1 FOLLOWER]: Refusing update from remote peer 28b576eb79094049811af6b44ebb2ec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:18:23.901207 19282 raft_consensus.cc:1261] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45 [term 1 FOLLOWER]: Refusing update from remote peer 28b576eb79094049811af6b44ebb2ec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:18:23.902158 19616 consensus_queue.cc:1035] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "232ce9e786c9454caabcb0bd2a4810cd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:18:23.902632 19614 consensus_queue.cc:1035] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0c11a59edbb4d7eb8da90fa20806a45" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:18:23.914301 19619 mvcc.cc:205] Tried to move back new op lower bound from 7298560425570082816 to 7298560425397387264. Current Snapshot: MvccSnapshot[applied={T|T < 7298560425570082816}]
I20260619 13:18:23.927309 19141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:18:23.927538 19141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:18:23.930696 19141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9459915fbbf4a6183324f3e2d14841b: Sending DeleteTablet for 3 replicas of tablet 9126a9efa5ec490abc3215d5dc07956c
I20260619 13:18:23.931833 19523 tablet_service.cc:1514] Processing DeleteTablet for tablet 9126a9efa5ec490abc3215d5dc07956c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:23 UTC) from {username='slave'} at 127.0.0.1:52590
I20260619 13:18:23.931833 19262 tablet_service.cc:1514] Processing DeleteTablet for tablet 9126a9efa5ec490abc3215d5dc07956c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:23 UTC) from {username='slave'} at 127.0.0.1:39942
I20260619 13:18:23.932017 19393 tablet_service.cc:1514] Processing DeleteTablet for tablet 9126a9efa5ec490abc3215d5dc07956c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:23 UTC) from {username='slave'} at 127.0.0.1:56236
I20260619 13:18:23.935118 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 19200
I20260619 13:18:23.948258 19628 tablet_replica.cc:330] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5: stopping tablet replica
I20260619 13:18:23.950464 19628 raft_consensus.cc:2229] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:18:23.951256 19628 raft_consensus.cc:2258] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:23.953449 19630 tablet_replica.cc:330] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd: stopping tablet replica
I20260619 13:18:23.953990 19629 tablet_replica.cc:330] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45: stopping tablet replica
I20260619 13:18:23.954207 19630 raft_consensus.cc:2229] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:18:23.954399 19629 raft_consensus.cc:2229] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:18:23.954777 19630 raft_consensus.cc:2258] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:23.954895 19629 raft_consensus.cc:2258] T 9126a9efa5ec490abc3215d5dc07956c P f0c11a59edbb4d7eb8da90fa20806a45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:23.955557 19628 ts_tablet_manager.cc:1905] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:23.955765 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:23.936489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.0.97.1:44299)
0619 13:18:23.936667 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:23.936674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:23.941287 (+ 4613us) server_negotiation.cc:407] Connection header received
0619 13:18:23.942195 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:23.942201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:23.942336 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:23.942430 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:23.943588 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.944735 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.948524 (+ 3789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:23.949200 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:23.949298 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:23.951815 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:23.951827 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:23.951844 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:23.952630 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:23.952694 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:23.952726 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:23.952808 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:23.955338 (+ 2530us) server_negotiation.cc:299] Negotiation successful
0619 13:18:23.955447 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:18:23.956861 19322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:23.957089 19630 ts_tablet_manager.cc:1905] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:23.963244 19628 ts_tablet_manager.cc:1918] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:23.963462 19628 log.cc:1196] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/9126a9efa5ec490abc3215d5dc07956c
I20260619 13:18:23.964010 19628 ts_tablet_manager.cc:1939] T 9126a9efa5ec490abc3215d5dc07956c P 28b576eb79094049811af6b44ebb2ec5: Deleting consensus metadata
I20260619 13:18:23.964335 19630 ts_tablet_manager.cc:1918] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:23.964565 19630 log.cc:1196] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/9126a9efa5ec490abc3215d5dc07956c
I20260619 13:18:23.965224 19630 ts_tablet_manager.cc:1939] T 9126a9efa5ec490abc3215d5dc07956c P 232ce9e786c9454caabcb0bd2a4810cd: Deleting consensus metadata
I20260619 13:18:23.966131 19125 catalog_manager.cc:4841] TS 28b576eb79094049811af6b44ebb2ec5 (127.0.97.3:39757): tablet 9126a9efa5ec490abc3215d5dc07956c (table test-table [id=9f1992fc528b483a906370c96abefd98]) successfully deleted
I20260619 13:18:23.967432 19125 catalog_manager.cc:4841] TS 232ce9e786c9454caabcb0bd2a4810cd (127.0.97.2:35923): tablet 9126a9efa5ec490abc3215d5dc07956c (table test-table [id=9f1992fc528b483a906370c96abefd98]) successfully deleted
I20260619 13:18:24.053661 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 19200
W20260619 13:18:24.074442 19128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44299 (error 108)
I20260619 13:18:24.074678 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 19331
I20260619 13:18:24.076375 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:24.075883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44299 (local address 127.0.0.1:39952)
0619 13:18:24.076075 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:24.076195 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 13:18:24.076628 19128 catalog_manager.cc:4584] TS f0c11a59edbb4d7eb8da90fa20806a45 (127.0.97.1:44299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9126a9efa5ec490abc3215d5dc07956c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44299: connect: Connection refused (error 111)
I20260619 13:18:24.082903 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:24.075695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.0.97.2:35923)
0619 13:18:24.075859 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:24.075865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:24.075994 (+ 129us) server_negotiation.cc:407] Connection header received
0619 13:18:24.076286 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:24.076292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:24.076407 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:24.076512 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:24.077099 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:24.078262 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:24.079560 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:24.080192 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:24.080317 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:24.081258 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:24.081270 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:24.081289 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:24.082194 (+ 905us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:24.082256 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:24.082290 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:24.082404 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:24.082480 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 13:18:24.082593 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 13:18:24.083652 19453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:24.130350 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:24.129811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44299 (local address 127.0.0.1:39966)
0619 13:18:24.130069 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:24.130178 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 13:18:24.131531 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:24.131138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44299 (local address 127.0.0.1:39972)
0619 13:18:24.131292 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:24.131379 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 13:18:24.170701 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 19331
I20260619 13:18:24.184870 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 19461
I20260619 13:18:24.191599 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:24.185608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.0.97.3:39757)
0619 13:18:24.185808 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:24.185817 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:24.185857 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:18:24.186034 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:24.186039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:24.186124 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:24.186219 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:24.186623 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:24.187417 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:24.188656 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:24.189101 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:24.189207 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:24.190236 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:24.190244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:24.190261 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:24.190851 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:24.190900 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:24.190921 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:24.191012 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:24.191156 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 13:18:24.191247 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:18:24.191882 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:24.191466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44299 (local address 127.0.0.1:39982)
0619 13:18:24.191649 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:24.191742 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 13:18:24.192231 19583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:24.193007 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:24.192570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44299 (local address 127.0.0.1:39996)
0619 13:18:24.192727 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:24.192826 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 13:18:24.273614 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:24.273014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44299 (local address 127.0.0.1:40000)
0619 13:18:24.273308 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:24.273440 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 13:18:24.274750 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:24.274279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44299 (local address 127.0.0.1:40014)
0619 13:18:24.274505 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:24.274598 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260619 13:18:24.307803 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 19461
I20260619 13:18:24.322841 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 19110
I20260619 13:18:24.323915 19171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:24.428972 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 19110
2026-06-19T13:18:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2750 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:24 dist-test-slave-mkt3 krb5kdc[19639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:24 dist-test-slave-mkt3 krb5kdc[19639](info): set up 2 sockets
Jun 19 13:18:24 dist-test-slave-mkt3 krb5kdc[19639](info): commencing operation
krb5kdc: starting...
W20260619 13:18:25.437803 388 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:25 dist-test-slave-mkt3 krb5kdc[19639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:18:25.658847 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44579
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:25.971889 19652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:25.972194 19652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:25.972332 19652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:25.979414 19652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:25.979527 19652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:25.979611 19652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:25.979687 19652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:25.979746 19652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:25.989404 19652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:25.990677 19652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:25.991940 19652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:26.003116 19658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:26.003105 19657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:26.003116 19660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:26.004153 19652 server_base.cc:1034] running on GCE node
I20260619 13:18:26.004878 19652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:26.006331 19652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:26.007543 19652 hybrid_clock.cc:648] HybridClock initialized: now 1781875106007492 us; error 32 us; skew 500 ppm
I20260619 13:18:26.007884 19652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:26 dist-test-slave-mkt3 krb5kdc[19639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:26.010869 19652 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:18:26.012333 19652 webserver.cc:460] Webserver started at http://127.0.97.62:43505/ using document root <none> and password file <none>
I20260619 13:18:26.013055 19652 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:26.013168 19652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:26.013451 19652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:26.016350 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "924b8f92582642dd8eb66099e12018aa"
format_stamp: "Formatted at 2026-06-19 13:18:26 on dist-test-slave-mkt3"
server_key: "3e1a197fea4b3ffdbe087ae1ad1d086a"
server_key_iv: "070faa6be42481dfeae97378bb10cdb6"
server_key_version: "encryptionkey@0"
I20260619 13:18:26.017082 19652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "924b8f92582642dd8eb66099e12018aa"
format_stamp: "Formatted at 2026-06-19 13:18:26 on dist-test-slave-mkt3"
server_key: "3e1a197fea4b3ffdbe087ae1ad1d086a"
server_key_iv: "070faa6be42481dfeae97378bb10cdb6"
server_key_version: "encryptionkey@0"
I20260619 13:18:26.022114 19652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:18:26.025389 19667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:26.026777 19652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:18:26.026942 19652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "924b8f92582642dd8eb66099e12018aa"
format_stamp: "Formatted at 2026-06-19 13:18:26 on dist-test-slave-mkt3"
server_key: "3e1a197fea4b3ffdbe087ae1ad1d086a"
server_key_iv: "070faa6be42481dfeae97378bb10cdb6"
server_key_version: "encryptionkey@0"
I20260619 13:18:26.027122 19652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:26.039383 19652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:26.043407 19652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:26.043663 19652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:26.060350 19652 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44579
I20260619 13:18:26.060381 19718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44579 every 8 connection(s)
I20260619 13:18:26.061682 19652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:26.065261 19719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:26.069895 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 19652
I20260619 13:18:26.070077 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:26.070639 388 external_mini_cluster.cc:1428] Setting key 14303355c06115d7942250cb87372240
I20260619 13:18:26.076081 19719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa: Bootstrap starting.
I20260619 13:18:26.079519 19719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:26.080649 19719 log.cc:824] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:26.083231 19719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa: No bootstrap required, opened a new log
Jun 19 13:18:26 dist-test-slave-mkt3 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:18:26.092183 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:26.072377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44432 (local address 127.0.97.62:44579)
0619 13:18:26.073109 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:26.073121 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:26.073164 (+ 43us) server_negotiation.cc:407] Connection header received
0619 13:18:26.074903 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:26.074926 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:26.075295 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:26.075604 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:26.077193 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:26.078559 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:26.080336 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:26.081026 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:26.081217 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:26.085333 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:26.085341 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:26.085358 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:26.085391 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:26.088342 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:26.089090 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:26.089095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:26.089104 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:26.089279 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:26.089667 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:26.089670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:26.089672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:26.090218 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:26.090426 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:26.090870 (+ 444us) server_negotiation.cc:299] Negotiation successful
0619 13:18:26.091221 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":330,"threads_started":1}
I20260619 13:18:26.092023 19719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924b8f92582642dd8eb66099e12018aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } }
I20260619 13:18:26.092375 19719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:26.092473 19719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924b8f92582642dd8eb66099e12018aa, State: Initialized, Role: FOLLOWER
I20260619 13:18:26.093030 19719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924b8f92582642dd8eb66099e12018aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } }
I20260619 13:18:26.093258 19719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:26.093425 19719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:26.093590 19719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:26.095752 19719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924b8f92582642dd8eb66099e12018aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } }
I20260619 13:18:26.096226 19719 leader_election.cc:302] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 924b8f92582642dd8eb66099e12018aa; no voters:
I20260619 13:18:26.096710 19719 leader_election.cc:288] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:26.096815 19724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:26.097427 19724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [term 1 LEADER]: Becoming Leader. State: Replica: 924b8f92582642dd8eb66099e12018aa, State: Running, Role: LEADER
I20260619 13:18:26.097960 19724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924b8f92582642dd8eb66099e12018aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } }
I20260619 13:18:26.098517 19719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:26.100823 19726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 924b8f92582642dd8eb66099e12018aa. Latest consensus state: current_term: 1 leader_uuid: "924b8f92582642dd8eb66099e12018aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924b8f92582642dd8eb66099e12018aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } } }
I20260619 13:18:26.101238 19726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:26.101042 19725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "924b8f92582642dd8eb66099e12018aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924b8f92582642dd8eb66099e12018aa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44579 } } }
I20260619 13:18:26.101380 19725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:26.102125 19730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:26.108469 19730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:26.119326 19730 catalog_manager.cc:1347] Generated new cluster ID: bfe911764bf34677889b5597033efc52
I20260619 13:18:26.119436 19730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:26.139003 19730 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:26.140873 19730 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:26.151824 19730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa: Generated new TSK 0
I20260619 13:18:26.152866 19730 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 13:18:26.208642 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44579
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:26.523666 19745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:26.523947 19745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:26.524072 19745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:26.531275 19745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:26.531409 19745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:26.531579 19745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:18:26.543459 19745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:26.544684 19745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:26.545969 19745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:26.557984 19750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:26.557934 19753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:26.557958 19751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:26.559007 19745 server_base.cc:1034] running on GCE node
I20260619 13:18:26.559540 19745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:26.560415 19745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:26.561621 19745 hybrid_clock.cc:648] HybridClock initialized: now 1781875106561565 us; error 44 us; skew 500 ppm
I20260619 13:18:26.561950 19745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:26 dist-test-slave-mkt3 krb5kdc[19639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:26.564934 19745 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 13:18:26.566341 19745 webserver.cc:460] Webserver started at http://127.0.97.1:44989/ using document root <none> and password file <none>
I20260619 13:18:26.566967 19745 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:26.567085 19745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:26.567360 19745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:26.570104 19745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "cfba3e8046574e489fa80aebb4bd6cd6"
format_stamp: "Formatted at 2026-06-19 13:18:26 on dist-test-slave-mkt3"
server_key: "89b0f3d30d2e063a4a23b67f198a28c3"
server_key_iv: "a8ec701647e94bb72f10b0a7deaf563a"
server_key_version: "encryptionkey@0"
I20260619 13:18:26.570820 19745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cfba3e8046574e489fa80aebb4bd6cd6"
format_stamp: "Formatted at 2026-06-19 13:18:26 on dist-test-slave-mkt3"
server_key: "89b0f3d30d2e063a4a23b67f198a28c3"
server_key_iv: "a8ec701647e94bb72f10b0a7deaf563a"
server_key_version: "encryptionkey@0"
I20260619 13:18:26.575497 19745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:18:26.578634 19760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:26.579829 19745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260619 13:18:26.579972 19745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "cfba3e8046574e489fa80aebb4bd6cd6"
format_stamp: "Formatted at 2026-06-19 13:18:26 on dist-test-slave-mkt3"
server_key: "89b0f3d30d2e063a4a23b67f198a28c3"
server_key_iv: "a8ec701647e94bb72f10b0a7deaf563a"
server_key_version: "encryptionkey@0"
I20260619 13:18:26.580157 19745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:26.590857 19745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:26.594785 19745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:26.595027 19745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:26.595862 19745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:26.597071 19745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:26.597144 19745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:26.597261 19745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:26.597313 19745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:26.626578 19745 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42175
I20260619 13:18:26.626617 19872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42175 every 8 connection(s)
I20260619 13:18:26.627779 19745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:18:26.633122 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 19745
I20260619 13:18:26.633266 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:18:26.633656 388 external_mini_cluster.cc:1428] Setting key a39ad9f927042c1060099c5533a002e9
Jun 19 13:18:26 dist-test-slave-mkt3 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:18:26.646459 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:26.630134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53221 (local address 127.0.97.62:44579)
0619 13:18:26.630544 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:26.630548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:26.631493 (+ 945us) server_negotiation.cc:407] Connection header received
0619 13:18:26.633323 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:26.633329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:26.633412 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:26.633503 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:26.635056 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:26.635804 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:26.637268 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:26.637651 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:26.637757 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:26.641002 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:26.641007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:26.641010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:26.641051 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:26.643433 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:26.644216 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:26.644220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:26.644223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:26.644339 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:26.644837 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:26.644841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:26.644843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:26.645138 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:26.645257 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:26.646038 (+ 781us) server_negotiation.cc:299] Negotiation successful
0619 13:18:26.646205 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":202,"threads_started":1}
I20260619 13:18:26.647686 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:26.630690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.1:53221)
0619 13:18:26.631351 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:26.631388 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:26.633057 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:26.633751 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:26.633762 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:26.634280 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:26.634904 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:26.634921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:26.635947 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:26.635951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:26.637136 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:26.637152 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:26.637875 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:26.637885 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:26.638109 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:26.638721 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:18:26.638742 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:26.640803 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:26.643559 (+ 2756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:26.643567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:26.643584 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:26.644086 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:26.644431 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:26.644435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:26.644439 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:26.644669 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:26.645254 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:26.645261 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:26.645651 (+ 390us) client_negotiation.cc:769] Sending connection context
0619 13:18:26.645948 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 13:18:26.646447 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":256,"threads_started":1}
I20260619 13:18:26.649125 19873 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44579
I20260619 13:18:26.649431 19873 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:26.650238 19873 heartbeater.cc:511] Master 127.0.97.62:44579 requested a full tablet report, sending...
I20260619 13:18:26.652411 19684 ts_manager.cc:194] Registered new tserver with Master: cfba3e8046574e489fa80aebb4bd6cd6 (127.0.97.1:42175)
I20260619 13:18:26.654029 19684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53221
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 13:18:26.684544 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44579
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:26.989943 19879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:26.990214 19879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:26.990346 19879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:26.997210 19879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:26.997319 19879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:26.997466 19879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:18:27.007067 19879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:27.008299 19879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:27.009598 19879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:27.021687 19887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:27.021733 19885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:27.021803 19884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:27.022399 19879 server_base.cc:1034] running on GCE node
I20260619 13:18:27.022930 19879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:27.023747 19879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:27.024928 19879 hybrid_clock.cc:648] HybridClock initialized: now 1781875107024873 us; error 46 us; skew 500 ppm
I20260619 13:18:27.025230 19879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:27 dist-test-slave-mkt3 krb5kdc[19639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:27.028126 19879 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 13:18:27.029464 19879 webserver.cc:460] Webserver started at http://127.0.97.2:42113/ using document root <none> and password file <none>
I20260619 13:18:27.030145 19879 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:27.030258 19879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:27.030540 19879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:27.033267 19879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "4c138dc9656b4d2ea80f935f950612d7"
format_stamp: "Formatted at 2026-06-19 13:18:27 on dist-test-slave-mkt3"
server_key: "28dd26d2bb76e6adf704b33bd8b5401b"
server_key_iv: "6ed1a01374415308b97dbc68a97ce30f"
server_key_version: "encryptionkey@0"
I20260619 13:18:27.034026 19879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4c138dc9656b4d2ea80f935f950612d7"
format_stamp: "Formatted at 2026-06-19 13:18:27 on dist-test-slave-mkt3"
server_key: "28dd26d2bb76e6adf704b33bd8b5401b"
server_key_iv: "6ed1a01374415308b97dbc68a97ce30f"
server_key_version: "encryptionkey@0"
I20260619 13:18:27.038789 19879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 13:18:27.042008 19894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:27.043213 19879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 13:18:27.043352 19879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "4c138dc9656b4d2ea80f935f950612d7"
format_stamp: "Formatted at 2026-06-19 13:18:27 on dist-test-slave-mkt3"
server_key: "28dd26d2bb76e6adf704b33bd8b5401b"
server_key_iv: "6ed1a01374415308b97dbc68a97ce30f"
server_key_version: "encryptionkey@0"
I20260619 13:18:27.043525 19879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:27.078796 19879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:27.082806 19879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:27.083066 19879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:27.084029 19879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:27.085461 19879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:27.085552 19879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:27.085668 19879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:27.085718 19879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:27.113750 19879 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45247
I20260619 13:18:27.113794 20006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45247 every 8 connection(s)
I20260619 13:18:27.115039 19879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:18:27.115885 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 19879
I20260619 13:18:27.116060 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:18:27.116586 388 external_mini_cluster.cc:1428] Setting key 02f70cf8915ccc87dd2e9911f29f6a31
Jun 19 13:18:27 dist-test-slave-mkt3 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:18:27.136544 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.117920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40633 (local address 127.0.97.62:44579)
0619 13:18:27.118072 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:27.118077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:27.119590 (+ 1513us) server_negotiation.cc:407] Connection header received
0619 13:18:27.121965 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:27.121969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:27.122034 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:27.122117 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:27.123866 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.124577 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.126666 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.127072 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.127176 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.130590 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:27.130596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:27.130601 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:27.130642 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:27.133533 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:27.134284 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:27.134292 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:27.134295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:27.134441 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:27.134835 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:27.134839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:27.134843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:27.135179 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:27.135342 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:27.136004 (+ 662us) server_negotiation.cc:299] Negotiation successful
0619 13:18:27.136227 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 13:18:27.137362 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.118567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.2:40633)
0619 13:18:27.119402 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:27.119463 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:27.121654 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:27.122366 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:27.122377 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:27.123004 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:27.123717 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:27.123741 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.124718 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.124724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:27.126495 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:27.126510 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.127282 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.127288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:27.127547 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.128271 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:18:27.128293 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:27.130370 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:27.133679 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:27.133686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:27.133699 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:27.134170 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:27.134542 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:27.134545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:27.134547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:27.134724 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:27.135335 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:27.135341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:27.135648 (+ 307us) client_negotiation.cc:769] Sending connection context
0619 13:18:27.135874 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 13:18:27.136276 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":309,"threads_started":1}
I20260619 13:18:27.138849 20007 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44579
I20260619 13:18:27.139175 20007 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:27.140058 20007 heartbeater.cc:511] Master 127.0.97.62:44579 requested a full tablet report, sending...
I20260619 13:18:27.141644 19684 ts_manager.cc:194] Registered new tserver with Master: 4c138dc9656b4d2ea80f935f950612d7 (127.0.97.2:45247)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 13:18:27.142521 19684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 13:18:27.170626 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44579
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:27.477492 20012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:27.477751 20012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:27.477919 20012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:27.485052 20012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:27.485162 20012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:27.485312 20012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:18:27.494648 20012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:27.495843 20012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:27.497116 20012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:27.509338 20017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:27.509424 20018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:27.509407 20020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:27.510581 20012 server_base.cc:1034] running on GCE node
I20260619 13:18:27.511137 20012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:27.512017 20012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:27.513208 20012 hybrid_clock.cc:648] HybridClock initialized: now 1781875107513151 us; error 50 us; skew 500 ppm
I20260619 13:18:27.513516 20012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:27 dist-test-slave-mkt3 krb5kdc[19639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:27.516534 20012 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 13:18:27.517795 20012 webserver.cc:460] Webserver started at http://127.0.97.3:43519/ using document root <none> and password file <none>
I20260619 13:18:27.518437 20012 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:27.518530 20012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:27.518783 20012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:27.521442 20012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "72792a32618144eeb2800855431cc8b0"
format_stamp: "Formatted at 2026-06-19 13:18:27 on dist-test-slave-mkt3"
server_key: "48b5c001623494b42991104222e03bba"
server_key_iv: "a6f902a016835b4b713dc537af048860"
server_key_version: "encryptionkey@0"
I20260619 13:18:27.522176 20012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "72792a32618144eeb2800855431cc8b0"
format_stamp: "Formatted at 2026-06-19 13:18:27 on dist-test-slave-mkt3"
server_key: "48b5c001623494b42991104222e03bba"
server_key_iv: "a6f902a016835b4b713dc537af048860"
server_key_version: "encryptionkey@0"
I20260619 13:18:27.526789 20012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 13:18:27.529942 20027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:27.531291 20012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:18:27.531424 20012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "72792a32618144eeb2800855431cc8b0"
format_stamp: "Formatted at 2026-06-19 13:18:27 on dist-test-slave-mkt3"
server_key: "48b5c001623494b42991104222e03bba"
server_key_iv: "a6f902a016835b4b713dc537af048860"
server_key_version: "encryptionkey@0"
I20260619 13:18:27.531585 20012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:27.546429 20012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:27.550107 20012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:27.550349 20012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:27.551203 20012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:27.552403 20012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:27.552472 20012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:27.552580 20012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:27.552628 20012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:27.580011 20012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33701
I20260619 13:18:27.580044 20139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33701 every 8 connection(s)
I20260619 13:18:27.581197 20012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:18:27.591224 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 20012
I20260619 13:18:27.591382 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:18:27.591797 388 external_mini_cluster.cc:1428] Setting key 629fea2b481ebe9e03bb3a6808ca1190
Jun 19 13:18:27 dist-test-slave-mkt3 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:18:27.599282 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.583536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59511 (local address 127.0.97.62:44579)
0619 13:18:27.583729 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:27.583735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:27.584845 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 13:18:27.586615 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:27.586621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:27.586687 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:27.586769 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:27.588079 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.588796 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.590264 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.590646 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.590736 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.594251 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:27.594257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:27.594262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:27.594300 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:27.596542 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:27.597142 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:27.597146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:27.597148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:27.597256 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:27.597670 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:27.597674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:27.597676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:27.598036 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:27.598177 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:27.598843 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 13:18:27.599013 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:18:27.600277 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.584009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44579 (local address 127.0.97.3:59511)
0619 13:18:27.584698 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:27.584741 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:27.586371 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:27.586941 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:27.586949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:27.587428 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:27.587956 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:27.587971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.588921 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.588924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:27.590154 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:27.590165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.590800 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.590804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:27.591002 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.591615 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:18:27.591636 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:27.594056 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:27.596661 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:27.596667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:27.596679 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:27.597054 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:27.597372 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:18:27.597375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:18:27.597377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:18:27.597535 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:18:27.598174 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:27.598181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:27.598485 (+ 304us) client_negotiation.cc:769] Sending connection context
0619 13:18:27.598731 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 13:18:27.599136 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":268,"threads_started":1}
I20260619 13:18:27.601496 20140 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44579
I20260619 13:18:27.601753 20140 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:27.602469 20140 heartbeater.cc:511] Master 127.0.97.62:44579 requested a full tablet report, sending...
I20260619 13:18:27.603852 19684 ts_manager.cc:194] Registered new tserver with Master: 72792a32618144eeb2800855431cc8b0 (127.0.97.3:33701)
I20260619 13:18:27.604645 19684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59511
I20260619 13:18:27.606758 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:18:27.621924 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.610501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44438 (local address 127.0.97.62:44579)
0619 13:18:27.610677 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:27.610683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:27.611022 (+ 339us) server_negotiation.cc:407] Connection header received
0619 13:18:27.611171 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:27.611175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:27.611230 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:27.611305 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:27.612584 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.613274 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.614552 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.614937 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.615023 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.617137 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:27.617141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:27.617144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:27.617173 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:27.619611 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:27.620223 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:27.620235 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:27.620238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:27.620351 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:27.620704 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:27.620707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:27.620709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:27.621003 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:27.621190 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:27.621460 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 13:18:27.621611 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:18:27.625385 19684 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:18:27.634377 19684 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:18:27.657440 19873 heartbeater.cc:503] Master 127.0.97.62:44579 was elected leader, sending a full tablet report...
I20260619 13:18:27.664307 20153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.645559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33701 (local address 127.0.0.1:51108)
0619 13:18:27.652541 (+ 6982us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:27.652583 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:27.652757 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:27.654910 (+ 2153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:27.654915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:27.654944 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:27.655487 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:27.655499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.658282 (+ 2783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.658288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:27.660931 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:27.660947 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.663214 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.663221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:27.663437 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.663491 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 13:18:27.663629 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 13:18:27.663842 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6744,"mutex_wait_us":89,"thread_start_us":7699,"threads_started":1}
I20260619 13:18:27.664489 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.645833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42175 (local address 127.0.0.1:33024)
0619 13:18:27.653502 (+ 7669us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:27.653518 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:27.653624 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:27.654702 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:27.654711 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:27.654743 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:27.655181 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:27.655212 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.657319 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.657326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:27.661311 (+ 3985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:27.661328 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.663543 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.663550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:27.663750 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.663786 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 13:18:27.663874 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 13:18:27.664017 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7536,"mutex_wait_us":2,"thread_start_us":6330,"threads_started":1}
I20260619 13:18:27.665409 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.646112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.0.97.3:33701)
0619 13:18:27.654009 (+ 7897us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:27.654018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:27.654053 (+ 35us) server_negotiation.cc:407] Connection header received
0619 13:18:27.654172 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:27.654180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:27.654432 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:27.654625 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:27.655707 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.658096 (+ 2389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.661162 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.662705 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.662831 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.663073 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:27.664856 (+ 1783us) server_negotiation.cc:299] Negotiation successful
0619 13:18:27.665043 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6984,"thread_start_us":6804,"threads_started":1}
I20260619 13:18:27.665423 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.646290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33024 (local address 127.0.97.1:42175)
0619 13:18:27.653761 (+ 7471us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:27.653770 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:27.653800 (+ 30us) server_negotiation.cc:407] Connection header received
0619 13:18:27.653915 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:27.653923 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:27.654187 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:27.654418 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:27.655422 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.657133 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.661502 (+ 4369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.663021 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.663144 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.663393 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:27.664896 (+ 1503us) server_negotiation.cc:299] Negotiation successful
0619 13:18:27.665080 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7322,"thread_start_us":7177,"threads_started":1}
I20260619 13:18:27.669448 20075 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5b7a0b58fc4151b69713d1450fb261 (DEFAULT_TABLE table=test-table [id=7f695dd8c3a94ebf922eb8ea46c2e3d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:27.669445 19808 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5b7a0b58fc4151b69713d1450fb261 (DEFAULT_TABLE table=test-table [id=7f695dd8c3a94ebf922eb8ea46c2e3d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:27.670826 20075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5b7a0b58fc4151b69713d1450fb261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:27.671238 19808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5b7a0b58fc4151b69713d1450fb261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:27.673265 20153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.653780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45247 (local address 127.0.0.1:50610)
0619 13:18:27.664574 (+ 10794us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:27.664594 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:27.664689 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:27.666933 (+ 2244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:27.666938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:27.666964 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:27.667111 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:27.667120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.668998 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.669003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:27.670790 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:27.670805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.672551 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.672557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:27.672752 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.672787 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 13:18:27.672860 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 13:18:27.672994 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10696,"thread_start_us":11974,"threads_started":1}
I20260619 13:18:27.674222 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.653916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50610 (local address 127.0.97.2:45247)
0619 13:18:27.666332 (+ 12416us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:27.666340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:27.666369 (+ 29us) server_negotiation.cc:407] Connection header received
0619 13:18:27.666466 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:27.666474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:27.666739 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:27.666933 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:27.667283 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.668850 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.670990 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.672375 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.673472 (+ 1097us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.673656 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:27.673740 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 13:18:27.673919 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12251,"thread_start_us":12118,"threads_started":1}
I20260619 13:18:27.677644 19942 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5b7a0b58fc4151b69713d1450fb261 (DEFAULT_TABLE table=test-table [id=7f695dd8c3a94ebf922eb8ea46c2e3d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:27.678982 19942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5b7a0b58fc4151b69713d1450fb261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:27.683194 20158 tablet_bootstrap.cc:492] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6: Bootstrap starting.
I20260619 13:18:27.685490 20159 tablet_bootstrap.cc:492] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0: Bootstrap starting.
I20260619 13:18:27.686931 20158 tablet_bootstrap.cc:654] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:27.688176 20158 log.cc:824] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:27.688896 20159 tablet_bootstrap.cc:654] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:27.689955 20159 log.cc:824] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:27.691105 20158 tablet_bootstrap.cc:492] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6: No bootstrap required, opened a new log
I20260619 13:18:27.691356 20158 ts_tablet_manager.cc:1397] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 13:18:27.692211 20162 tablet_bootstrap.cc:492] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7: Bootstrap starting.
I20260619 13:18:27.692844 20159 tablet_bootstrap.cc:492] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0: No bootstrap required, opened a new log
I20260619 13:18:27.693106 20159 ts_tablet_manager.cc:1397] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 13:18:27.695068 20162 tablet_bootstrap.cc:654] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:27.696074 20162 log.cc:824] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:27.698415 20162 tablet_bootstrap.cc:492] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7: No bootstrap required, opened a new log
I20260619 13:18:27.698626 20162 ts_tablet_manager.cc:1397] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 13:18:27.701794 20158 raft_consensus.cc:348] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72792a32618144eeb2800855431cc8b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33701 } } peers { permanent_uuid: "cfba3e8046574e489fa80aebb4bd6cd6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42175 } } peers { permanent_uuid: "4c138dc9656b4d2ea80f935f950612d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45247 } }
I20260619 13:18:27.702225 20158 raft_consensus.cc:374] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:27.702316 20158 raft_consensus.cc:729] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfba3e8046574e489fa80aebb4bd6cd6, State: Initialized, Role: FOLLOWER
I20260619 13:18:27.702912 20158 consensus_queue.cc:260] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72792a32618144eeb2800855431cc8b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33701 } } peers { permanent_uuid: "cfba3e8046574e489fa80aebb4bd6cd6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42175 } } peers { permanent_uuid: "4c138dc9656b4d2ea80f935f950612d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45247 } }
I20260619 13:18:27.702979 20159 raft_consensus.cc:348] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72792a32618144eeb2800855431cc8b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33701 } } peers { permanent_uuid: "cfba3e8046574e489fa80aebb4bd6cd6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42175 } } peers { permanent_uuid: "4c138dc9656b4d2ea80f935f950612d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45247 } }
I20260619 13:18:27.703356 20159 raft_consensus.cc:374] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:27.703454 20159 raft_consensus.cc:729] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72792a32618144eeb2800855431cc8b0, State: Initialized, Role: FOLLOWER
I20260619 13:18:27.704135 20159 consensus_queue.cc:260] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72792a32618144eeb2800855431cc8b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33701 } } peers { permanent_uuid: "cfba3e8046574e489fa80aebb4bd6cd6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42175 } } peers { permanent_uuid: "4c138dc9656b4d2ea80f935f950612d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45247 } }
I20260619 13:18:27.707778 20162 raft_consensus.cc:348] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72792a32618144eeb2800855431cc8b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33701 } } peers { permanent_uuid: "cfba3e8046574e489fa80aebb4bd6cd6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42175 } } peers { permanent_uuid: "4c138dc9656b4d2ea80f935f950612d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45247 } }
I20260619 13:18:27.708115 20162 raft_consensus.cc:374] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:27.708181 20162 raft_consensus.cc:729] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c138dc9656b4d2ea80f935f950612d7, State: Initialized, Role: FOLLOWER
I20260619 13:18:27.708762 20162 consensus_queue.cc:260] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72792a32618144eeb2800855431cc8b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33701 } } peers { permanent_uuid: "cfba3e8046574e489fa80aebb4bd6cd6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42175 } } peers { permanent_uuid: "4c138dc9656b4d2ea80f935f950612d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45247 } }
I20260619 13:18:27.711654 20158 ts_tablet_manager.cc:1428] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6: Time spent starting tablet: real 0.020s user 0.013s sys 0.008s
I20260619 13:18:27.712386 20140 heartbeater.cc:503] Master 127.0.97.62:44579 was elected leader, sending a full tablet report...
I20260619 13:18:27.713029 20159 ts_tablet_manager.cc:1428] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0: Time spent starting tablet: real 0.020s user 0.008s sys 0.012s
I20260619 13:18:27.713721 20007 heartbeater.cc:503] Master 127.0.97.62:44579 was elected leader, sending a full tablet report...
I20260619 13:18:27.714596 20162 ts_tablet_manager.cc:1428] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260619 13:18:27.789148 20166 raft_consensus.cc:482] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:18:27.789455 20166 raft_consensus.cc:504] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72792a32618144eeb2800855431cc8b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33701 } } peers { permanent_uuid: "cfba3e8046574e489fa80aebb4bd6cd6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42175 } } peers { permanent_uuid: "4c138dc9656b4d2ea80f935f950612d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45247 } }
I20260619 13:18:27.791155 20166 leader_election.cc:288] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72792a32618144eeb2800855431cc8b0 (127.0.97.3:33701), cfba3e8046574e489fa80aebb4bd6cd6 (127.0.97.1:42175)
I20260619 13:18:27.797817 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.791422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60143 (local address 127.0.97.3:33701)
0619 13:18:27.791632 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:27.791639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:27.792403 (+ 764us) server_negotiation.cc:407] Connection header received
0619 13:18:27.792554 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:27.792560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:27.792637 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:27.792719 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:27.793364 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.794282 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.796115 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.796894 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.796991 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.797093 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:27.797445 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 13:18:27.797583 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 13:18:27.797974 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.791322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33701 (local address 127.0.97.2:60143)
0619 13:18:27.792272 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:27.792292 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:27.792422 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:27.792744 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:27.792748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:27.792780 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:27.793144 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:27.793157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.794412 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.794418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:27.795961 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:27.795973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.797118 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.797125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:27.797281 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.797320 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 13:18:27.797391 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 13:18:27.797510 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":25,"thread_start_us":543,"threads_started":1}
I20260619 13:18:27.798003 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.791579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42175 (local address 127.0.97.2:36213)
0619 13:18:27.792383 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:27.792401 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:27.792532 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:27.792902 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:27.792906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:27.792992 (+ 86us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:27.793411 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:27.793421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.794581 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.794585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:27.796008 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:27.796029 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.797154 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.797158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:27.797293 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.797340 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 13:18:27.797421 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 13:18:27.797544 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":40,"spinlock_wait_cycles":25984,"thread_start_us":503,"threads_started":1}
I20260619 13:18:27.797833 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.791667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36213 (local address 127.0.97.1:42175)
0619 13:18:27.791937 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:27.791942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:27.792498 (+ 556us) server_negotiation.cc:407] Connection header received
0619 13:18:27.792663 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:27.792668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:27.792739 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:27.792823 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:27.793553 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.794432 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.796170 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.796916 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.797049 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.797144 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:27.797464 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 13:18:27.797583 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260619 13:18:27.798823 20095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5b7a0b58fc4151b69713d1450fb261" candidate_uuid: "4c138dc9656b4d2ea80f935f950612d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72792a32618144eeb2800855431cc8b0" is_pre_election: true
I20260619 13:18:27.799165 19828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5b7a0b58fc4151b69713d1450fb261" candidate_uuid: "4c138dc9656b4d2ea80f935f950612d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfba3e8046574e489fa80aebb4bd6cd6" is_pre_election: true
I20260619 13:18:27.799337 20095 raft_consensus.cc:2454] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c138dc9656b4d2ea80f935f950612d7 in term 0.
I20260619 13:18:27.799649 19828 raft_consensus.cc:2454] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c138dc9656b4d2ea80f935f950612d7 in term 0.
I20260619 13:18:27.800020 19895 leader_election.cc:302] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c138dc9656b4d2ea80f935f950612d7, 72792a32618144eeb2800855431cc8b0; no voters:
I20260619 13:18:27.800421 20166 raft_consensus.cc:2781] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:18:27.800526 20166 raft_consensus.cc:482] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:18:27.800612 20166 raft_consensus.cc:3037] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:27.802801 20166 raft_consensus.cc:504] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72792a32618144eeb2800855431cc8b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33701 } } peers { permanent_uuid: "cfba3e8046574e489fa80aebb4bd6cd6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42175 } } peers { permanent_uuid: "4c138dc9656b4d2ea80f935f950612d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45247 } }
I20260619 13:18:27.803411 20166 leader_election.cc:288] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [CANDIDATE]: Term 1 election: Requested vote from peers 72792a32618144eeb2800855431cc8b0 (127.0.97.3:33701), cfba3e8046574e489fa80aebb4bd6cd6 (127.0.97.1:42175)
I20260619 13:18:27.803807 20095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5b7a0b58fc4151b69713d1450fb261" candidate_uuid: "4c138dc9656b4d2ea80f935f950612d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72792a32618144eeb2800855431cc8b0"
I20260619 13:18:27.803844 19828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5b7a0b58fc4151b69713d1450fb261" candidate_uuid: "4c138dc9656b4d2ea80f935f950612d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfba3e8046574e489fa80aebb4bd6cd6"
I20260619 13:18:27.804003 20095 raft_consensus.cc:3037] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:27.804056 19828 raft_consensus.cc:3037] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:27.806176 19828 raft_consensus.cc:2454] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c138dc9656b4d2ea80f935f950612d7 in term 1.
I20260619 13:18:27.806176 20095 raft_consensus.cc:2454] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c138dc9656b4d2ea80f935f950612d7 in term 1.
I20260619 13:18:27.806541 19895 leader_election.cc:302] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c138dc9656b4d2ea80f935f950612d7, 72792a32618144eeb2800855431cc8b0; no voters:
I20260619 13:18:27.806784 20166 raft_consensus.cc:2781] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:27.807168 20166 raft_consensus.cc:686] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [term 1 LEADER]: Becoming Leader. State: Replica: 4c138dc9656b4d2ea80f935f950612d7, State: Running, Role: LEADER
I20260619 13:18:27.807591 20166 consensus_queue.cc:237] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72792a32618144eeb2800855431cc8b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33701 } } peers { permanent_uuid: "cfba3e8046574e489fa80aebb4bd6cd6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42175 } } peers { permanent_uuid: "4c138dc9656b4d2ea80f935f950612d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45247 } }
I20260619 13:18:27.812139 19684 catalog_manager.cc:5495] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c138dc9656b4d2ea80f935f950612d7 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "4c138dc9656b4d2ea80f935f950612d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72792a32618144eeb2800855431cc8b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfba3e8046574e489fa80aebb4bd6cd6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c138dc9656b4d2ea80f935f950612d7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45247 } health_report { overall_health: HEALTHY } } }
W20260619 13:18:27.832677 20141 tablet.cc:2369] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:18:27.833922 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.829076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50612 (local address 127.0.97.2:45247)
0619 13:18:27.829233 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:27.829237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:27.829252 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:18:27.829339 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:27.829342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:27.829420 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:27.829532 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:18:27.829819 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.830558 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.831931 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.832346 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.832441 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.832950 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:18:27.833426 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:18:27.833529 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:27.833625 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 13:18:27.833685 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 13:18:27.841331 20095 raft_consensus.cc:1261] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0 [term 1 FOLLOWER]: Refusing update from remote peer 4c138dc9656b4d2ea80f935f950612d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:18:27.841333 19828 raft_consensus.cc:1261] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6 [term 1 FOLLOWER]: Refusing update from remote peer 4c138dc9656b4d2ea80f935f950612d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:18:27.842290 20169 consensus_queue.cc:1035] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72792a32618144eeb2800855431cc8b0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:18:27.842583 20166 consensus_queue.cc:1035] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfba3e8046574e489fa80aebb4bd6cd6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:18:27.854485 20173 mvcc.cc:205] Tried to move back new op lower bound from 7298560441709461504 to 7298560441583882240. Current Snapshot: MvccSnapshot[applied={T|T < 7298560441709461504}]
I20260619 13:18:27.860086 20174 mvcc.cc:205] Tried to move back new op lower bound from 7298560441709461504 to 7298560441583882240. Current Snapshot: MvccSnapshot[applied={T|T < 7298560441709461504}]
I20260619 13:18:27.865960 19684 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:18:27.866209 19684 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:18:27.869978 19684 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 924b8f92582642dd8eb66099e12018aa: Sending DeleteTablet for 3 replicas of tablet 1a5b7a0b58fc4151b69713d1450fb261
I20260619 13:18:27.871099 20075 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5b7a0b58fc4151b69713d1450fb261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51108
I20260619 13:18:27.871358 19942 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5b7a0b58fc4151b69713d1450fb261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50610
I20260619 13:18:27.871596 19808 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5b7a0b58fc4151b69713d1450fb261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33024
I20260619 13:18:27.873575 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 19745
I20260619 13:18:27.881764 20184 tablet_replica.cc:330] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6: stopping tablet replica
I20260619 13:18:27.882614 20182 tablet_replica.cc:330] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0: stopping tablet replica
I20260619 13:18:27.882805 20184 raft_consensus.cc:2229] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:18:27.883016 20182 raft_consensus.cc:2229] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:18:27.883335 20184 raft_consensus.cc:2258] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:27.883561 20182 raft_consensus.cc:2258] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:27.884065 20183 tablet_replica.cc:330] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7: stopping tablet replica
I20260619 13:18:27.884442 20183 raft_consensus.cc:2229] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:18:27.885124 20183 raft_consensus.cc:2258] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:27.886351 20182 ts_tablet_manager.cc:1905] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:27.886974 20184 ts_tablet_manager.cc:1905] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:27.891494 20183 ts_tablet_manager.cc:1905] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:27.892926 20182 ts_tablet_manager.cc:1918] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:27.893069 20182 log.cc:1196] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/1a5b7a0b58fc4151b69713d1450fb261
I20260619 13:18:27.893543 20182 ts_tablet_manager.cc:1939] T 1a5b7a0b58fc4151b69713d1450fb261 P 72792a32618144eeb2800855431cc8b0: Deleting consensus metadata
I20260619 13:18:27.893776 20184 ts_tablet_manager.cc:1918] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:27.893956 20184 log.cc:1196] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-0/wal/wals/1a5b7a0b58fc4151b69713d1450fb261
I20260619 13:18:27.894423 20184 ts_tablet_manager.cc:1939] T 1a5b7a0b58fc4151b69713d1450fb261 P cfba3e8046574e489fa80aebb4bd6cd6: Deleting consensus metadata
I20260619 13:18:27.895376 19668 catalog_manager.cc:4841] TS 72792a32618144eeb2800855431cc8b0 (127.0.97.3:33701): tablet 1a5b7a0b58fc4151b69713d1450fb261 (table test-table [id=7f695dd8c3a94ebf922eb8ea46c2e3d7]) successfully deleted
I20260619 13:18:27.895998 19669 catalog_manager.cc:4841] TS cfba3e8046574e489fa80aebb4bd6cd6 (127.0.97.1:42175): tablet 1a5b7a0b58fc4151b69713d1450fb261 (table test-table [id=7f695dd8c3a94ebf922eb8ea46c2e3d7]) successfully deleted
Jun 19 13:18:27 dist-test-slave-mkt3 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 13:18:27.901757 20183 ts_tablet_manager.cc:1918] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:27.901988 20183 log.cc:1196] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/1a5b7a0b58fc4151b69713d1450fb261
I20260619 13:18:27.902518 20183 ts_tablet_manager.cc:1939] T 1a5b7a0b58fc4151b69713d1450fb261 P 4c138dc9656b4d2ea80f935f950612d7: Deleting consensus metadata
I20260619 13:18:27.903230 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:27.874668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33034 (local address 127.0.97.1:42175)
0619 13:18:27.874840 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:27.874846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:27.881975 (+ 7129us) server_negotiation.cc:407] Connection header received
0619 13:18:27.882167 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:27.882174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:27.882273 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:27.882377 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:27.884975 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.886155 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.888182 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:27.888853 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:27.888951 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:27.897407 (+ 8456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:27.897419 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:27.897432 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:27.897465 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:27.900299 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:27.900952 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:27.900961 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:27.900965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:27.901112 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:27.901455 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:27.901460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:27.901464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:27.901798 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:27.902138 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:27.902702 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 13:18:27.902901 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:18:27.903834 19868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:27.904253 19668 catalog_manager.cc:4841] TS 4c138dc9656b4d2ea80f935f950612d7 (127.0.97.2:45247): tablet 1a5b7a0b58fc4151b69713d1450fb261 (table test-table [id=7f695dd8c3a94ebf922eb8ea46c2e3d7]) successfully deleted
I20260619 13:18:27.990670 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 19745
I20260619 13:18:28.005601 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 19879
Jun 19 13:18:28 dist-test-slave-mkt3 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 13:18:28.022185 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:28.006527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50614 (local address 127.0.97.2:45247)
0619 13:18:28.006723 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:28.006730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:28.006751 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:18:28.006867 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:28.006872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:28.006966 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:28.007073 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:28.008308 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:28.009501 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:28.010841 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:28.011488 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:28.011612 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:28.015661 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:28.015680 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:28.015701 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:28.015755 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:28.019466 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:28.020112 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:28.020121 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:28.020124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:28.020284 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:28.020580 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:28.020585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:28.020588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:28.021078 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:28.021391 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:28.021537 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 13:18:28.021773 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:18:28.022888 20002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:28.132714 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 19879
I20260619 13:18:28.147858 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 20012
Jun 19 13:18:28 dist-test-slave-mkt3 krb5kdc[19639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 13:18:28.162142 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:28.148656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.0.97.3:33701)
0619 13:18:28.148832 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:28.148837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:28.148856 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:18:28.149029 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:28.149035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:28.149100 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:28.149202 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:28.150486 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:28.151221 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:28.152461 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:28.152848 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:28.152954 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:28.157110 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:28.157127 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:28.157146 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:28.157183 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:28.159817 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:28.160463 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:28.160470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:28.160472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:28.160612 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:28.160892 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:28.160895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:28.160897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:28.161247 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:28.161461 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:28.161651 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 13:18:28.161831 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:18:28.162747 20135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:28.255759 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 20012
I20260619 13:18:28.273440 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 19652
I20260619 13:18:28.274468 19714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:28.381522 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 19652
2026-06-19T13:18:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (3953 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T13:18:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:28Z Disabled control of system clock
I20260619 13:18:28.449432 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39097
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:18:28.761312 20193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:28.761575 20193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:28.761654 20193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:28.769200 20193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:28.769332 20193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:28.769397 20193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:28.769474 20193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:28.769531 20193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:28.779072 20193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:28.780282 20193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:28.782073 20193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:28.792754 20201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:28.792820 20198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:28.792788 20199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:28.793790 20193 server_base.cc:1034] running on GCE node
I20260619 13:18:28.794495 20193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:28.795748 20193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:28.796952 20193 hybrid_clock.cc:648] HybridClock initialized: now 1781875108796904 us; error 41 us; skew 500 ppm
I20260619 13:18:28.797272 20193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:18:28.799335 20193 webserver.cc:460] Webserver started at http://127.0.97.62:42317/ using document root <none> and password file <none>
I20260619 13:18:28.799966 20193 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:28.800063 20193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:28.800354 20193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:28.803158 20193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "cc873398792e4e06afeb3c0183b06600"
format_stamp: "Formatted at 2026-06-19 13:18:28 on dist-test-slave-mkt3"
server_key: "7c88b291f5739cf9e8b17daae7c9a802"
server_key_iv: "a1339af7ffcc2b6650f3d602fc41a63d"
server_key_version: "encryptionkey@0"
I20260619 13:18:28.803898 20193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "cc873398792e4e06afeb3c0183b06600"
format_stamp: "Formatted at 2026-06-19 13:18:28 on dist-test-slave-mkt3"
server_key: "7c88b291f5739cf9e8b17daae7c9a802"
server_key_iv: "a1339af7ffcc2b6650f3d602fc41a63d"
server_key_version: "encryptionkey@0"
I20260619 13:18:28.808748 20193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 13:18:28.812047 20207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:28.813315 20193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:18:28.813463 20193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "cc873398792e4e06afeb3c0183b06600"
format_stamp: "Formatted at 2026-06-19 13:18:28 on dist-test-slave-mkt3"
server_key: "7c88b291f5739cf9e8b17daae7c9a802"
server_key_iv: "a1339af7ffcc2b6650f3d602fc41a63d"
server_key_version: "encryptionkey@0"
I20260619 13:18:28.813643 20193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:28.838423 20193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:28.839221 20193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:28.839478 20193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:28.857102 20193 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39097
I20260619 13:18:28.857125 20258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39097 every 8 connection(s)
I20260619 13:18:28.858434 20193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:28.858850 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 20193
I20260619 13:18:28.859009 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:28.859537 388 external_mini_cluster.cc:1428] Setting key 56a298bbdf59b6d3c29b5780cde38228
I20260619 13:18:28.862596 20259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:28.872208 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:28.861501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.0.97.62:39097)
0619 13:18:28.862464 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:28.862475 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:28.862516 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:18:28.864136 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:28.864157 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:28.864609 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:28.864981 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:28.865502 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:28.866844 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:28.868164 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:28.868813 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:28.868980 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:28.870046 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:28.870054 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:28.870069 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:28.870613 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:28.870685 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:28.870705 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:28.870808 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:28.871010 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 13:18:28.871333 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":403,"threads_started":1}
I20260619 13:18:28.872670 20259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600: Bootstrap starting.
I20260619 13:18:28.875841 20259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:28.876727 20259 log.cc:824] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:28.879281 20259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600: No bootstrap required, opened a new log
I20260619 13:18:28.886437 20259 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc873398792e4e06afeb3c0183b06600" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39097 } }
I20260619 13:18:28.886802 20259 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:28.886893 20259 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc873398792e4e06afeb3c0183b06600, State: Initialized, Role: FOLLOWER
I20260619 13:18:28.887447 20259 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc873398792e4e06afeb3c0183b06600" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39097 } }
I20260619 13:18:28.887657 20259 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:28.887778 20259 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:28.887919 20259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:28.890012 20259 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc873398792e4e06afeb3c0183b06600" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39097 } }
I20260619 13:18:28.890475 20259 leader_election.cc:302] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc873398792e4e06afeb3c0183b06600; no voters:
I20260619 13:18:28.891014 20259 leader_election.cc:288] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:28.891175 20264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:28.891919 20264 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [term 1 LEADER]: Becoming Leader. State: Replica: cc873398792e4e06afeb3c0183b06600, State: Running, Role: LEADER
I20260619 13:18:28.892419 20264 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc873398792e4e06afeb3c0183b06600" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39097 } }
I20260619 13:18:28.892963 20259 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:28.896468 20265 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc873398792e4e06afeb3c0183b06600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc873398792e4e06afeb3c0183b06600" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39097 } } }
I20260619 13:18:28.896868 20265 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:28.897251 20266 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc873398792e4e06afeb3c0183b06600. Latest consensus state: current_term: 1 leader_uuid: "cc873398792e4e06afeb3c0183b06600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc873398792e4e06afeb3c0183b06600" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39097 } } }
I20260619 13:18:28.897604 20266 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:28.898253 20273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:28.904065 20273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:28.913503 20273 catalog_manager.cc:1347] Generated new cluster ID: 50f58fe2d2c84ba9b6c347185bd2a509
I20260619 13:18:28.913602 20273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:28.927547 20273 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:28.928758 20273 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:28.939277 20273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600: Generated new TSK 0
I20260619 13:18:28.940075 20273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:28.946831 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39097
--builtin_ntp_servers=127.0.97.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 13:18:29.277673 20283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:29.277974 20283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:29.278102 20283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:29.285203 20283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:29.285301 20283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:29.285434 20283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 13:18:29.294620 20283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:29.295926 20283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:29.297799 20283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:29.310000 20289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:29.310436 20291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:29.310091 20288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:29.311014 20283 server_base.cc:1034] running on GCE node
I20260619 13:18:29.311656 20283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:29.312570 20283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:29.313760 20283 hybrid_clock.cc:648] HybridClock initialized: now 1781875109313711 us; error 43 us; skew 500 ppm
I20260619 13:18:29.314079 20283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:18:29.316113 20283 webserver.cc:460] Webserver started at http://127.0.97.1:35757/ using document root <none> and password file <none>
I20260619 13:18:29.316834 20283 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:29.316931 20283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:29.317224 20283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:29.320196 20283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/data/instance:
uuid: "a7c6cfd08d01472eb637016ebae42fe1"
format_stamp: "Formatted at 2026-06-19 13:18:29 on dist-test-slave-mkt3"
server_key: "413e1e2c22e97b5e0d0d4da0523e1387"
server_key_iv: "4fb9055cd63e064901d3040aa004ffa9"
server_key_version: "encryptionkey@0"
I20260619 13:18:29.320972 20283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a7c6cfd08d01472eb637016ebae42fe1"
format_stamp: "Formatted at 2026-06-19 13:18:29 on dist-test-slave-mkt3"
server_key: "413e1e2c22e97b5e0d0d4da0523e1387"
server_key_iv: "4fb9055cd63e064901d3040aa004ffa9"
server_key_version: "encryptionkey@0"
I20260619 13:18:29.326097 20283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 13:18:29.329553 20297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:29.330909 20283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:18:29.331061 20283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/wal
uuid: "a7c6cfd08d01472eb637016ebae42fe1"
format_stamp: "Formatted at 2026-06-19 13:18:29 on dist-test-slave-mkt3"
server_key: "413e1e2c22e97b5e0d0d4da0523e1387"
server_key_iv: "4fb9055cd63e064901d3040aa004ffa9"
server_key_version: "encryptionkey@0"
I20260619 13:18:29.331267 20283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:29.359860 20283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:29.360741 20283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:29.360988 20283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:29.362033 20283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:29.363567 20283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:29.363663 20283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:29.363772 20283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:29.363838 20283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:29.393071 20283 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33661
I20260619 13:18:29.393122 20409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33661 every 8 connection(s)
I20260619 13:18:29.394430 20283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/data/info.pb
I20260619 13:18:29.401378 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 20283
I20260619 13:18:29.401552 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-0/wal/instance
I20260619 13:18:29.402030 388 external_mini_cluster.cc:1428] Setting key 6b14340608c351742727678a781439ad
I20260619 13:18:29.404798 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39097
--builtin_ntp_servers=127.0.97.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:18:29.407325 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:29.396856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36451 (local address 127.0.97.62:39097)
0619 13:18:29.397274 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:29.397278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:29.398547 (+ 1269us) server_negotiation.cc:407] Connection header received
0619 13:18:29.400310 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:29.400315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:29.400393 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:29.400485 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:29.401760 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:29.402530 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:29.404465 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:29.405064 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:29.405193 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:29.406468 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:29.406475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:29.406479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:29.406499 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:29.406555 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:29.406569 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:29.406655 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:29.406834 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 13:18:29.406932 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":246,"threads_started":1}
I20260619 13:18:29.408253 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:29.397389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39097 (local address 127.0.97.1:36451)
0619 13:18:29.398399 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:29.398435 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:29.400036 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:29.400721 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:29.400731 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:29.400817 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:29.401563 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:29.401596 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:29.402712 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:29.402716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:29.404310 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:29.404323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:29.405261 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:29.405266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:29.405453 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:29.406148 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:18:29.406150 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:29.406202 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:18:29.406278 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:29.406648 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:29.406654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:29.406660 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 13:18:29.406808 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 13:18:29.407145 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":238,"threads_started":1}
I20260619 13:18:29.409688 20410 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39097
I20260619 13:18:29.410036 20410 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:29.410864 20410 heartbeater.cc:511] Master 127.0.97.62:39097 requested a full tablet report, sending...
I20260619 13:18:29.413093 20224 ts_manager.cc:194] Registered new tserver with Master: a7c6cfd08d01472eb637016ebae42fe1 (127.0.97.1:33661)
I20260619 13:18:29.414619 20224 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:36451
W20260619 13:18:29.713518 20414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:29.713770 20414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:29.713896 20414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:29.720956 20414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:29.721083 20414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:29.721235 20414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 13:18:29.730002 20414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:29.731165 20414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:29.732823 20414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:29.744833 20419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:29.744757 20422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:29.744796 20420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:29.745774 20414 server_base.cc:1034] running on GCE node
I20260619 13:18:29.746331 20414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:29.747211 20414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:29.748418 20414 hybrid_clock.cc:648] HybridClock initialized: now 1781875109748377 us; error 33 us; skew 500 ppm
I20260619 13:18:29.748704 20414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:18:29.757062 20414 webserver.cc:460] Webserver started at http://127.0.97.2:39659/ using document root <none> and password file <none>
I20260619 13:18:29.757748 20414 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:29.757881 20414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:29.758203 20414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:29.761004 20414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/data/instance:
uuid: "af795c01e2cd471c9951b4c53c7069f3"
format_stamp: "Formatted at 2026-06-19 13:18:29 on dist-test-slave-mkt3"
server_key: "e8954e6759844bbeea139483be377e82"
server_key_iv: "d2e0df70a2e660245767cafd005f7f52"
server_key_version: "encryptionkey@0"
I20260619 13:18:29.761782 20414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/wal/instance:
uuid: "af795c01e2cd471c9951b4c53c7069f3"
format_stamp: "Formatted at 2026-06-19 13:18:29 on dist-test-slave-mkt3"
server_key: "e8954e6759844bbeea139483be377e82"
server_key_iv: "d2e0df70a2e660245767cafd005f7f52"
server_key_version: "encryptionkey@0"
I20260619 13:18:29.766703 20414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:18:29.770244 20428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:29.771785 20414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:18:29.771952 20414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/wal
uuid: "af795c01e2cd471c9951b4c53c7069f3"
format_stamp: "Formatted at 2026-06-19 13:18:29 on dist-test-slave-mkt3"
server_key: "e8954e6759844bbeea139483be377e82"
server_key_iv: "d2e0df70a2e660245767cafd005f7f52"
server_key_version: "encryptionkey@0"
I20260619 13:18:29.772162 20414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:29.788535 20414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:29.789317 20414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:29.789563 20414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:29.790496 20414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:29.791874 20414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:29.791970 20414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:29.792068 20414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:29.792137 20414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:29.821388 20414 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41933
I20260619 13:18:29.821440 20540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41933 every 8 connection(s)
I20260619 13:18:29.822706 20414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/data/info.pb
I20260619 13:18:29.828084 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 20414
I20260619 13:18:29.828226 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/wal/instance
I20260619 13:18:29.828627 388 external_mini_cluster.cc:1428] Setting key c2bf644d73ae6194c039bea9941d54a8
I20260619 13:18:29.831190 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39097
--builtin_ntp_servers=127.0.97.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 13:18:29.834563 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:29.825126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55207 (local address 127.0.97.62:39097)
0619 13:18:29.825312 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:29.825316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:29.826637 (+ 1321us) server_negotiation.cc:407] Connection header received
0619 13:18:29.828363 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:29.828369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:29.828446 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:29.828541 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:29.829688 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:29.830526 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:29.832031 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:29.832419 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:29.832533 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:29.833770 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:29.833775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:29.833778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:29.833795 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:29.833826 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:29.833861 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:29.833939 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:29.834170 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 13:18:29.834263 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:18:29.835520 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:29.825654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39097 (local address 127.0.97.2:55207)
0619 13:18:29.826494 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:29.826530 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:29.828080 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:29.828784 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:29.828793 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:29.828880 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:29.829518 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:29.829539 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:29.830667 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:29.830671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:29.831900 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:29.831914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:29.832615 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:29.832622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:29.832830 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:29.833481 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:18:29.833484 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:29.833521 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:18:29.833595 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:29.833937 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:29.833948 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:29.833956 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 13:18:29.834128 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 13:18:29.834487 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":256,"threads_started":1}
I20260619 13:18:29.836766 20541 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39097
I20260619 13:18:29.837019 20541 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:29.837694 20541 heartbeater.cc:511] Master 127.0.97.62:39097 requested a full tablet report, sending...
I20260619 13:18:29.838938 20224 ts_manager.cc:194] Registered new tserver with Master: af795c01e2cd471c9951b4c53c7069f3 (127.0.97.2:41933)
I20260619 13:18:29.839890 20224 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:55207
W20260619 13:18:30.138041 20544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:30.138322 20544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:30.138448 20544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:30.145608 20544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:30.145720 20544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:30.145895 20544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 13:18:30.155125 20544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:30.156417 20544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:30.158215 20544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:30.170616 20552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:30.170642 20549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:30.170567 20550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:30.171415 20544 server_base.cc:1034] running on GCE node
I20260619 13:18:30.172000 20544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:30.172945 20544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:30.174136 20544 hybrid_clock.cc:648] HybridClock initialized: now 1781875110174083 us; error 50 us; skew 500 ppm
I20260619 13:18:30.174425 20544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:18:30.176628 20544 webserver.cc:460] Webserver started at http://127.0.97.3:33405/ using document root <none> and password file <none>
I20260619 13:18:30.177279 20544 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:30.177376 20544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:30.177667 20544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:30.180393 20544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/data/instance:
uuid: "75b5ce2d6ff7495582c69303cd46c0ad"
format_stamp: "Formatted at 2026-06-19 13:18:30 on dist-test-slave-mkt3"
server_key: "0f0f083839d1a25bf61daa85810b8282"
server_key_iv: "d0c58087e766ac9080ae9052beeb3ac9"
server_key_version: "encryptionkey@0"
I20260619 13:18:30.181149 20544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/wal/instance:
uuid: "75b5ce2d6ff7495582c69303cd46c0ad"
format_stamp: "Formatted at 2026-06-19 13:18:30 on dist-test-slave-mkt3"
server_key: "0f0f083839d1a25bf61daa85810b8282"
server_key_iv: "d0c58087e766ac9080ae9052beeb3ac9"
server_key_version: "encryptionkey@0"
I20260619 13:18:30.186036 20544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 13:18:30.189188 20558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:30.190435 20544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:18:30.190575 20544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/wal
uuid: "75b5ce2d6ff7495582c69303cd46c0ad"
format_stamp: "Formatted at 2026-06-19 13:18:30 on dist-test-slave-mkt3"
server_key: "0f0f083839d1a25bf61daa85810b8282"
server_key_iv: "d0c58087e766ac9080ae9052beeb3ac9"
server_key_version: "encryptionkey@0"
I20260619 13:18:30.190766 20544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:30.210461 20544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:30.211270 20544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:30.211520 20544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:30.212492 20544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 13:18:30.213943 20544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 13:18:30.214037 20544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:30.214139 20544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 13:18:30.214206 20544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:30.242962 20544 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36351
I20260619 13:18:30.242995 20670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36351 every 8 connection(s)
I20260619 13:18:30.244201 20544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/data/info.pb
I20260619 13:18:30.253257 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 20544
I20260619 13:18:30.253437 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/wal/instance
I20260619 13:18:30.253877 388 external_mini_cluster.cc:1428] Setting key 2525221213fb8871dc3780afab21a8a8
I20260619 13:18:30.256314 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.246638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45143 (local address 127.0.97.62:39097)
0619 13:18:30.246840 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:30.246846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:30.248201 (+ 1355us) server_negotiation.cc:407] Connection header received
0619 13:18:30.249905 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:30.249909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:30.249977 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:30.250061 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:30.251050 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.251769 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.253457 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.253914 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.254022 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.255523 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:30.255528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:30.255532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:30.255548 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:30.255581 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:30.255592 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:30.255676 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:30.255875 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 13:18:30.255967 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:18:30.257206 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.247187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39097 (local address 127.0.97.3:45143)
0619 13:18:30.248057 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:30.248092 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:30.249631 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:30.250253 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:30.250261 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:30.250348 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:18:30.250911 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:30.250926 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.251904 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.251908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:30.253303 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:30.253318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.254138 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.254146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:30.254401 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.255113 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 13:18:30.255116 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:18:30.255174 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 13:18:30.255324 (+ 150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:18:30.255687 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:18:30.255695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:18:30.255702 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 13:18:30.255842 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 13:18:30.256146 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":243,"threads_started":1}
I20260619 13:18:30.258466 20671 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39097
I20260619 13:18:30.258724 20671 heartbeater.cc:464] Registering TS with master...
I20260619 13:18:30.259397 20671 heartbeater.cc:511] Master 127.0.97.62:39097 requested a full tablet report, sending...
I20260619 13:18:30.260637 20224 ts_manager.cc:194] Registered new tserver with Master: 75b5ce2d6ff7495582c69303cd46c0ad (127.0.97.3:36351)
I20260619 13:18:30.261394 20224 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:45143
I20260619 13:18:30.268675 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 13:18:30.277892 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.272342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.0.97.62:39097)
0619 13:18:30.272511 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:30.272517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:30.272865 (+ 348us) server_negotiation.cc:407] Connection header received
0619 13:18:30.273017 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:30.273021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:30.273091 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:30.273169 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:30.273674 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.274385 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.275740 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.276157 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.276265 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.277223 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:30.277228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:30.277231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:30.277248 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:30.277278 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:30.277289 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:30.277360 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:30.277490 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 13:18:30.277580 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:18:30.281239 20224 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 13:18:30.287606 20224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 13:18:30.320536 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.299770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.0.97.2:41933)
0619 13:18:30.312446 (+ 12676us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:30.312453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:30.312477 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:18:30.312556 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:30.312561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:30.312742 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:30.312913 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:30.313510 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.315021 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.317778 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.319388 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.319513 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.319776 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:30.320013 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 13:18:30.320172 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12539,"thread_start_us":12426,"threads_started":1}
I20260619 13:18:30.320570 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.299197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41933 (local address 127.0.0.1:53394)
0619 13:18:30.310462 (+ 11265us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:30.310482 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:30.310601 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:30.312937 (+ 2336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:30.312943 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:30.312973 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:30.313302 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:30.313315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.315141 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.315149 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:30.317501 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:30.317516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.319569 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.319575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:30.319777 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.319827 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 13:18:30.319953 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 13:18:30.320125 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11115,"mutex_wait_us":24,"thread_start_us":12436,"threads_started":1}
I20260619 13:18:30.321053 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.300076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36351 (local address 127.0.0.1:60086)
0619 13:18:30.311450 (+ 11374us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:30.311466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:30.311567 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:30.314553 (+ 2986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:30.314559 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:30.314584 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:30.314988 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:30.314999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.316497 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.316502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:30.318537 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:30.318552 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.320228 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.320234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:30.320495 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.320531 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 13:18:30.320598 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 13:18:30.320711 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11250,"mutex_wait_us":1,"thread_start_us":10071,"threads_started":1}
I20260619 13:18:30.321667 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.298628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33661 (local address 127.0.0.1:48178)
0619 13:18:30.299511 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:30.299551 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:30.299718 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:30.314801 (+ 15083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:30.314806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:30.314836 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:30.315297 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:30.315308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.317362 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.317367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:30.319123 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:30.319134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.320883 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.320889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:30.321084 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.321115 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 13:18:30.321202 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 13:18:30.321319 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":4,"spinlock_wait_cycles":3968,"thread_start_us":611,"threads_started":1}
I20260619 13:18:30.322366 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.300602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.0.97.3:36351)
0619 13:18:30.313603 (+ 13001us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:30.313612 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:30.313644 (+ 32us) server_negotiation.cc:407] Connection header received
0619 13:18:30.313736 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:30.313743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:30.314354 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:30.315167 (+ 813us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:30.315373 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.316348 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.318743 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.320016 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.321373 (+ 1357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.321651 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:30.321719 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 13:18:30.322011 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12841,"thread_start_us":12693,"threads_started":1}
I20260619 13:18:30.322367 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.298860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.0.97.1:33661)
0619 13:18:30.314244 (+ 15384us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:30.314249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:30.314270 (+ 21us) server_negotiation.cc:407] Connection header received
0619 13:18:30.314358 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:30.314365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:30.314593 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:30.315518 (+ 925us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:30.315749 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.317234 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.319290 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.320664 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.320828 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.321057 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:30.321907 (+ 850us) server_negotiation.cc:299] Negotiation successful
0619 13:18:30.322050 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15258,"thread_start_us":15131,"threads_started":1}
I20260619 13:18:30.324980 20476 tablet_service.cc:1467] Processing CreateTablet for tablet 3349229aa39a4ac68b6e4ed77f23f37a (DEFAULT_TABLE table=test-table [id=7c91955dabae430983208efa6227c494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:30.325768 20606 tablet_service.cc:1467] Processing CreateTablet for tablet 3349229aa39a4ac68b6e4ed77f23f37a (DEFAULT_TABLE table=test-table [id=7c91955dabae430983208efa6227c494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:30.326503 20476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3349229aa39a4ac68b6e4ed77f23f37a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:30.327754 20606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3349229aa39a4ac68b6e4ed77f23f37a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:30.327925 20345 tablet_service.cc:1467] Processing CreateTablet for tablet 3349229aa39a4ac68b6e4ed77f23f37a (DEFAULT_TABLE table=test-table [id=7c91955dabae430983208efa6227c494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 13:18:30.328963 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3349229aa39a4ac68b6e4ed77f23f37a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:30.339425 20689 tablet_bootstrap.cc:492] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1: Bootstrap starting.
I20260619 13:18:30.341421 20690 tablet_bootstrap.cc:492] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3: Bootstrap starting.
I20260619 13:18:30.344141 20689 tablet_bootstrap.cc:654] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:30.344563 20690 tablet_bootstrap.cc:654] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:30.345347 20689 log.cc:824] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:30.345659 20690 log.cc:824] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:30.347009 20691 tablet_bootstrap.cc:492] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad: Bootstrap starting.
I20260619 13:18:30.348371 20689 tablet_bootstrap.cc:492] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1: No bootstrap required, opened a new log
I20260619 13:18:30.348532 20690 tablet_bootstrap.cc:492] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3: No bootstrap required, opened a new log
I20260619 13:18:30.348598 20689 ts_tablet_manager.cc:1397] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260619 13:18:30.348785 20690 ts_tablet_manager.cc:1397] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260619 13:18:30.350296 20691 tablet_bootstrap.cc:654] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:30.351368 20691 log.cc:824] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:30.354167 20691 tablet_bootstrap.cc:492] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad: No bootstrap required, opened a new log
I20260619 13:18:30.354409 20691 ts_tablet_manager.cc:1397] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 13:18:30.355898 20689 raft_consensus.cc:348] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c6cfd08d01472eb637016ebae42fe1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33661 } } peers { permanent_uuid: "75b5ce2d6ff7495582c69303cd46c0ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36351 } } peers { permanent_uuid: "af795c01e2cd471c9951b4c53c7069f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41933 } }
I20260619 13:18:30.356227 20689 raft_consensus.cc:374] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:30.356319 20689 raft_consensus.cc:729] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c6cfd08d01472eb637016ebae42fe1, State: Initialized, Role: FOLLOWER
I20260619 13:18:30.356848 20689 consensus_queue.cc:260] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c6cfd08d01472eb637016ebae42fe1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33661 } } peers { permanent_uuid: "75b5ce2d6ff7495582c69303cd46c0ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36351 } } peers { permanent_uuid: "af795c01e2cd471c9951b4c53c7069f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41933 } }
I20260619 13:18:30.357817 20410 heartbeater.cc:503] Master 127.0.97.62:39097 was elected leader, sending a full tablet report...
I20260619 13:18:30.358376 20689 ts_tablet_manager.cc:1428] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1: Time spent starting tablet: real 0.010s user 0.009s sys 0.002s
I20260619 13:18:30.359476 20690 raft_consensus.cc:348] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c6cfd08d01472eb637016ebae42fe1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33661 } } peers { permanent_uuid: "75b5ce2d6ff7495582c69303cd46c0ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36351 } } peers { permanent_uuid: "af795c01e2cd471c9951b4c53c7069f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41933 } }
I20260619 13:18:30.359897 20690 raft_consensus.cc:374] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:30.359997 20690 raft_consensus.cc:729] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af795c01e2cd471c9951b4c53c7069f3, State: Initialized, Role: FOLLOWER
I20260619 13:18:30.360540 20690 consensus_queue.cc:260] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c6cfd08d01472eb637016ebae42fe1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33661 } } peers { permanent_uuid: "75b5ce2d6ff7495582c69303cd46c0ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36351 } } peers { permanent_uuid: "af795c01e2cd471c9951b4c53c7069f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41933 } }
I20260619 13:18:30.361550 20541 heartbeater.cc:503] Master 127.0.97.62:39097 was elected leader, sending a full tablet report...
I20260619 13:18:30.362233 20690 ts_tablet_manager.cc:1428] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3: Time spent starting tablet: real 0.013s user 0.005s sys 0.009s
I20260619 13:18:30.366166 20691 raft_consensus.cc:348] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c6cfd08d01472eb637016ebae42fe1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33661 } } peers { permanent_uuid: "75b5ce2d6ff7495582c69303cd46c0ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36351 } } peers { permanent_uuid: "af795c01e2cd471c9951b4c53c7069f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41933 } }
I20260619 13:18:30.366462 20691 raft_consensus.cc:374] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:30.366540 20691 raft_consensus.cc:729] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5ce2d6ff7495582c69303cd46c0ad, State: Initialized, Role: FOLLOWER
I20260619 13:18:30.367023 20691 consensus_queue.cc:260] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c6cfd08d01472eb637016ebae42fe1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33661 } } peers { permanent_uuid: "75b5ce2d6ff7495582c69303cd46c0ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36351 } } peers { permanent_uuid: "af795c01e2cd471c9951b4c53c7069f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41933 } }
I20260619 13:18:30.367888 20671 heartbeater.cc:503] Master 127.0.97.62:39097 was elected leader, sending a full tablet report...
I20260619 13:18:30.368414 20691 ts_tablet_manager.cc:1428] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
W20260619 13:18:30.396454 20411 tablet.cc:2369] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:18:30.495728 20672 tablet.cc:2369] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 13:18:30.574627 20542 tablet.cc:2369] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:18:30.576195 20696 raft_consensus.cc:482] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 13:18:30.576460 20696 raft_consensus.cc:504] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c6cfd08d01472eb637016ebae42fe1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33661 } } peers { permanent_uuid: "75b5ce2d6ff7495582c69303cd46c0ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36351 } } peers { permanent_uuid: "af795c01e2cd471c9951b4c53c7069f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41933 } }
I20260619 13:18:30.578162 20696 leader_election.cc:288] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7c6cfd08d01472eb637016ebae42fe1 (127.0.97.1:33661), 75b5ce2d6ff7495582c69303cd46c0ad (127.0.97.3:36351)
I20260619 13:18:30.584647 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.578507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57419 (local address 127.0.97.1:33661)
0619 13:18:30.578725 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:30.578732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:30.579083 (+ 351us) server_negotiation.cc:407] Connection header received
0619 13:18:30.579320 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:30.579326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:30.579414 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:30.579518 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:30.580078 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.580964 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.582866 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.583733 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.583823 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.583889 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:30.584275 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 13:18:30.584392 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:18:30.584671 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.578402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33661 (local address 127.0.97.2:57419)
0619 13:18:30.578982 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:30.578997 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:30.579167 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:30.579542 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:30.579546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:30.579577 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:30.579930 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:30.579942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.581089 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.581095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:30.582691 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:30.582703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.583909 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.583916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:30.584105 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.584151 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 13:18:30.584254 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 13:18:30.584379 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":1,"thread_start_us":293,"threads_started":1}
I20260619 13:18:30.585004 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.579096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36351 (local address 127.0.97.2:47415)
0619 13:18:30.579700 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:30.579717 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:18:30.579828 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:18:30.580118 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:18:30.580122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:18:30.580148 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 13:18:30.580506 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:30.580517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.581617 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.581621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:30.583100 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:18:30.583114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.584342 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.584348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:18:30.584514 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.584547 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 13:18:30.584649 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 13:18:30.584765 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":316,"threads_started":1}
I20260619 13:18:30.585011 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.579221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47415 (local address 127.0.97.3:36351)
0619 13:18:30.579423 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:30.579429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:30.579797 (+ 368us) server_negotiation.cc:407] Connection header received
0619 13:18:30.579928 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:30.579933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:30.580003 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:30.580084 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 13:18:30.580642 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.581481 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.583285 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.584179 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.584280 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.584370 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:30.584686 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 13:18:30.584811 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:18:30.585939 20626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3349229aa39a4ac68b6e4ed77f23f37a" candidate_uuid: "af795c01e2cd471c9951b4c53c7069f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5ce2d6ff7495582c69303cd46c0ad" is_pre_election: true
I20260619 13:18:30.585937 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3349229aa39a4ac68b6e4ed77f23f37a" candidate_uuid: "af795c01e2cd471c9951b4c53c7069f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c6cfd08d01472eb637016ebae42fe1" is_pre_election: true
I20260619 13:18:30.586385 20365 raft_consensus.cc:2454] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af795c01e2cd471c9951b4c53c7069f3 in term 0.
I20260619 13:18:30.586385 20626 raft_consensus.cc:2454] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af795c01e2cd471c9951b4c53c7069f3 in term 0.
I20260619 13:18:30.586939 20430 leader_election.cc:302] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7c6cfd08d01472eb637016ebae42fe1, af795c01e2cd471c9951b4c53c7069f3; no voters:
I20260619 13:18:30.587263 20696 raft_consensus.cc:2781] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 13:18:30.587404 20696 raft_consensus.cc:482] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 13:18:30.587509 20696 raft_consensus.cc:3037] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:30.589541 20696 raft_consensus.cc:504] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c6cfd08d01472eb637016ebae42fe1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33661 } } peers { permanent_uuid: "75b5ce2d6ff7495582c69303cd46c0ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36351 } } peers { permanent_uuid: "af795c01e2cd471c9951b4c53c7069f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41933 } }
I20260619 13:18:30.590139 20696 leader_election.cc:288] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [CANDIDATE]: Term 1 election: Requested vote from peers a7c6cfd08d01472eb637016ebae42fe1 (127.0.97.1:33661), 75b5ce2d6ff7495582c69303cd46c0ad (127.0.97.3:36351)
I20260619 13:18:30.590500 20365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3349229aa39a4ac68b6e4ed77f23f37a" candidate_uuid: "af795c01e2cd471c9951b4c53c7069f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c6cfd08d01472eb637016ebae42fe1"
I20260619 13:18:30.590605 20626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3349229aa39a4ac68b6e4ed77f23f37a" candidate_uuid: "af795c01e2cd471c9951b4c53c7069f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5ce2d6ff7495582c69303cd46c0ad"
I20260619 13:18:30.590693 20365 raft_consensus.cc:3037] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:30.590791 20626 raft_consensus.cc:3037] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:30.593778 20626 raft_consensus.cc:2454] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af795c01e2cd471c9951b4c53c7069f3 in term 1.
I20260619 13:18:30.593778 20365 raft_consensus.cc:2454] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af795c01e2cd471c9951b4c53c7069f3 in term 1.
I20260619 13:18:30.594179 20430 leader_election.cc:302] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7c6cfd08d01472eb637016ebae42fe1, af795c01e2cd471c9951b4c53c7069f3; no voters:
I20260619 13:18:30.594470 20696 raft_consensus.cc:2781] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:30.594862 20696 raft_consensus.cc:686] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [term 1 LEADER]: Becoming Leader. State: Replica: af795c01e2cd471c9951b4c53c7069f3, State: Running, Role: LEADER
I20260619 13:18:30.595360 20696 consensus_queue.cc:237] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c6cfd08d01472eb637016ebae42fe1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33661 } } peers { permanent_uuid: "75b5ce2d6ff7495582c69303cd46c0ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36351 } } peers { permanent_uuid: "af795c01e2cd471c9951b4c53c7069f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41933 } }
I20260619 13:18:30.599821 20223 catalog_manager.cc:5495] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to af795c01e2cd471c9951b4c53c7069f3 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "af795c01e2cd471c9951b4c53c7069f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c6cfd08d01472eb637016ebae42fe1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75b5ce2d6ff7495582c69303cd46c0ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af795c01e2cd471c9951b4c53c7069f3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41933 } health_report { overall_health: HEALTHY } } }
I20260619 13:18:30.683293 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.678173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.0.97.2:41933)
0619 13:18:30.678384 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:30.678390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:30.678408 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:18:30.678590 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:30.678593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:30.678691 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:30.678819 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:18:30.679159 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.679860 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.681250 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.681656 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.681761 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.682326 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:18:30.682815 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:18:30.682937 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:30.682979 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 13:18:30.683041 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 13:18:30.691097 20365 raft_consensus.cc:1261] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1 [term 1 FOLLOWER]: Refusing update from remote peer af795c01e2cd471c9951b4c53c7069f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:18:30.691133 20626 raft_consensus.cc:1261] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad [term 1 FOLLOWER]: Refusing update from remote peer af795c01e2cd471c9951b4c53c7069f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:18:30.691964 20696 consensus_queue.cc:1035] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5ce2d6ff7495582c69303cd46c0ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:18:30.692266 20700 consensus_queue.cc:1035] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c6cfd08d01472eb637016ebae42fe1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:18:30.702901 20705 mvcc.cc:205] Tried to move back new op lower bound from 7298560453380718592 to 7298560453002301440. Current Snapshot: MvccSnapshot[applied={T|T < 7298560453380718592}]
I20260619 13:18:30.704473 20704 mvcc.cc:205] Tried to move back new op lower bound from 7298560453380718592 to 7298560453002301440. Current Snapshot: MvccSnapshot[applied={T|T < 7298560453380718592}]
I20260619 13:18:30.714740 20224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:18:30.714965 20224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 13:18:30.718246 20224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc873398792e4e06afeb3c0183b06600: Sending DeleteTablet for 3 replicas of tablet 3349229aa39a4ac68b6e4ed77f23f37a
I20260619 13:18:30.719329 20345 tablet_service.cc:1514] Processing DeleteTablet for tablet 3349229aa39a4ac68b6e4ed77f23f37a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:30 UTC) from {username='slave'} at 127.0.0.1:48178
I20260619 13:18:30.719329 20606 tablet_service.cc:1514] Processing DeleteTablet for tablet 3349229aa39a4ac68b6e4ed77f23f37a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:30 UTC) from {username='slave'} at 127.0.0.1:60086
I20260619 13:18:30.719380 20476 tablet_service.cc:1514] Processing DeleteTablet for tablet 3349229aa39a4ac68b6e4ed77f23f37a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 13:18:30 UTC) from {username='slave'} at 127.0.0.1:53394
I20260619 13:18:30.723459 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 20283
I20260619 13:18:30.731762 20715 tablet_replica.cc:330] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3: stopping tablet replica
I20260619 13:18:30.732218 20715 raft_consensus.cc:2229] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 13:18:30.732870 20715 raft_consensus.cc:2258] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:30.733566 20714 tablet_replica.cc:330] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1: stopping tablet replica
I20260619 13:18:30.733932 20714 raft_consensus.cc:2229] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:18:30.734398 20714 raft_consensus.cc:2258] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:30.736358 20715 ts_tablet_manager.cc:1905] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:30.736531 20714 ts_tablet_manager.cc:1905] T 3349229aa39a4ac68b6e4ed77f23f37a P a7c6cfd08d01472eb637016ebae42fe1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:30.739027 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.724739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.0.97.1:33661)
0619 13:18:30.724909 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:30.724915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:30.727861 (+ 2946us) server_negotiation.cc:407] Connection header received
0619 13:18:30.728037 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:30.728042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:30.728164 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:30.728277 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:30.732004 (+ 3727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.733144 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.735099 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.735776 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.735872 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.737363 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:30.737374 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:30.737390 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:30.738133 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:30.738199 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:30.738228 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:30.738327 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:30.738549 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 13:18:30.738655 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:18:30.740056 20405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:30.746634 20715 ts_tablet_manager.cc:1918] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:30.746807 20715 log.cc:1196] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-1/wal/wals/3349229aa39a4ac68b6e4ed77f23f37a
I20260619 13:18:30.747342 20715 ts_tablet_manager.cc:1939] T 3349229aa39a4ac68b6e4ed77f23f37a P af795c01e2cd471c9951b4c53c7069f3: Deleting consensus metadata
I20260619 13:18:30.749155 20211 catalog_manager.cc:4841] TS af795c01e2cd471c9951b4c53c7069f3 (127.0.97.2:41933): tablet 3349229aa39a4ac68b6e4ed77f23f37a (table test-table [id=7c91955dabae430983208efa6227c494]) successfully deleted
I20260619 13:18:30.749872 20713 tablet_replica.cc:330] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad: stopping tablet replica
I20260619 13:18:30.750196 20713 raft_consensus.cc:2229] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 13:18:30.750604 20713 raft_consensus.cc:2258] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 13:18:30.752743 20713 ts_tablet_manager.cc:1905] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 13:18:30.759424 20713 ts_tablet_manager.cc:1918] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 13:18:30.759573 20713 log.cc:1196] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad: Deleting WAL directory at /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781874807295266-388-0/minicluster-data/ts-2/wal/wals/3349229aa39a4ac68b6e4ed77f23f37a
I20260619 13:18:30.760017 20713 ts_tablet_manager.cc:1939] T 3349229aa39a4ac68b6e4ed77f23f37a P 75b5ce2d6ff7495582c69303cd46c0ad: Deleting consensus metadata
I20260619 13:18:30.761700 20209 catalog_manager.cc:4841] TS 75b5ce2d6ff7495582c69303cd46c0ad (127.0.97.3:36351): tablet 3349229aa39a4ac68b6e4ed77f23f37a (table test-table [id=7c91955dabae430983208efa6227c494]) successfully deleted
I20260619 13:18:30.851349 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 20283
W20260619 13:18:30.869540 20209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33661 (error 108)
I20260619 13:18:30.870478 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 20414
I20260619 13:18:30.879335 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.878517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33661 (local address 127.0.0.1:48194)
0619 13:18:30.878979 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:30.879129 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"threads_started":1}
W20260619 13:18:30.879765 20209 catalog_manager.cc:4584] TS a7c6cfd08d01472eb637016ebae42fe1 (127.0.97.1:33661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3349229aa39a4ac68b6e4ed77f23f37a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33661: connect: Connection refused (error 111)
I20260619 13:18:30.880980 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.871558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53418 (local address 127.0.97.2:41933)
0619 13:18:30.871727 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:30.871733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:30.871962 (+ 229us) server_negotiation.cc:407] Connection header received
0619 13:18:30.872224 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:30.872230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:30.872330 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:30.872411 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:30.873067 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.874423 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.876437 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:30.877119 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:30.877224 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:30.879034 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:30.879045 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:30.879065 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:30.880057 (+ 992us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:30.880122 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:30.880146 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:30.880270 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:30.880392 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 13:18:30.880487 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:18:30.882299 20536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:30.933414 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.932872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33661 (local address 127.0.0.1:48200)
0619 13:18:30.933128 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:30.933246 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 13:18:30.934530 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.934124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33661 (local address 127.0.0.1:48212)
0619 13:18:30.934284 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:30.934369 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 13:18:30.994078 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 20414
I20260619 13:18:30.995677 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.995189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33661 (local address 127.0.0.1:48214)
0619 13:18:30.995456 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:30.995534 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260619 13:18:30.996765 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:30.996339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33661 (local address 127.0.0.1:48218)
0619 13:18:30.996497 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:30.996590 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 13:18:31.009778 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 20544
I20260619 13:18:31.015980 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:31.010528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60094 (local address 127.0.97.3:36351)
0619 13:18:31.010695 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:31.010700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:31.010732 (+ 32us) server_negotiation.cc:407] Connection header received
0619 13:18:31.010893 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:31.010897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:31.010987 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:31.011067 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:31.011467 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:31.012175 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:31.013402 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:31.013778 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:31.013889 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:31.014770 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:31.014778 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:31.014792 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:31.015318 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:31.015362 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:31.015384 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:31.015470 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:31.015574 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 13:18:31.015654 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:18:31.016589 20666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:31.077803 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:31.077315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33661 (local address 127.0.0.1:48224)
0619 13:18:31.077539 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:31.077658 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 13:18:31.078918 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:31.078508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33661 (local address 127.0.0.1:48232)
0619 13:18:31.078718 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 13:18:31.078787 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 13:18:31.102960 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 20544
I20260619 13:18:31.115943 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 20193
I20260619 13:18:31.117035 20254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:31.222395 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 20193
2026-06-19T13:18:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2840 ms)
[----------] 6 tests from EncryptionPolicyTest (20418 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:31 dist-test-slave-mkt3 krb5kdc[20725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:31 dist-test-slave-mkt3 krb5kdc[20725](info): set up 2 sockets
Jun 19 13:18:31 dist-test-slave-mkt3 krb5kdc[20725](info): commencing operation
krb5kdc: starting...
W20260619 13:18:32.226086 388 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:32 dist-test-slave-mkt3 krb5kdc[20725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:18:32.445574 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:32.755924 20738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:32.756223 20738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:32.756359 20738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:32.763826 20738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:32.763945 20738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:32.764017 20738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:32.764096 20738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:32.764153 20738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:32.773707 20738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:32.774966 20738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:32.776207 20738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:32.787849 20743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:32.787899 20744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:32.788179 20746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:32.789014 20738 server_base.cc:1034] running on GCE node
I20260619 13:18:32.789781 20738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:32.791177 20738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:32.792392 20738 hybrid_clock.cc:648] HybridClock initialized: now 1781875112792341 us; error 40 us; skew 500 ppm
I20260619 13:18:32.792739 20738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:32 dist-test-slave-mkt3 krb5kdc[20725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:32.795692 20738 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:18:32.797084 20738 webserver.cc:460] Webserver started at http://127.0.0.1:35281/ using document root <none> and password file <none>
I20260619 13:18:32.797742 20738 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:32.797879 20738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:32.798166 20738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:32.801026 20738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "1c53542cd1254cbeb6926beaa3e3ce2c"
format_stamp: "Formatted at 2026-06-19 13:18:32 on dist-test-slave-mkt3"
server_key: "34c7cf9ae1882fa0b367b5d1736e9f97"
server_key_iv: "39d6b6915dbaf38df0e8327ad5d9706a"
server_key_version: "encryptionkey@0"
I20260619 13:18:32.801745 20738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "1c53542cd1254cbeb6926beaa3e3ce2c"
format_stamp: "Formatted at 2026-06-19 13:18:32 on dist-test-slave-mkt3"
server_key: "34c7cf9ae1882fa0b367b5d1736e9f97"
server_key_iv: "39d6b6915dbaf38df0e8327ad5d9706a"
server_key_version: "encryptionkey@0"
I20260619 13:18:32.806795 20738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:18:32.810396 20753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:32.811650 20738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:18:32.811802 20738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "1c53542cd1254cbeb6926beaa3e3ce2c"
format_stamp: "Formatted at 2026-06-19 13:18:32 on dist-test-slave-mkt3"
server_key: "34c7cf9ae1882fa0b367b5d1736e9f97"
server_key_iv: "39d6b6915dbaf38df0e8327ad5d9706a"
server_key_version: "encryptionkey@0"
I20260619 13:18:32.811970 20738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:32.835127 20738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:32.838838 20738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:32.839080 20738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:32.855245 20738 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39249
I20260619 13:18:32.855278 20804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39249 every 8 connection(s)
I20260619 13:18:32.856511 20738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:32.860164 20805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:32.866686 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 20738
I20260619 13:18:32.866884 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:32.867512 388 external_mini_cluster.cc:1428] Setting key 1eede5b0cba2058a994d9ffb5944b5bd
I20260619 13:18:32.870237 20805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c: Bootstrap starting.
I20260619 13:18:32.873474 20805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:32.874636 20805 log.cc:824] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:32.877943 20805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c: No bootstrap required, opened a new log
Jun 19 13:18:32 dist-test-slave-mkt3 krb5kdc[20725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 13:18:32.886664 20805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c53542cd1254cbeb6926beaa3e3ce2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39249 } }
I20260619 13:18:32.887137 20805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:32.887248 20805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c53542cd1254cbeb6926beaa3e3ce2c, State: Initialized, Role: FOLLOWER
I20260619 13:18:32.888010 20805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c53542cd1254cbeb6926beaa3e3ce2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39249 } }
I20260619 13:18:32.888310 20805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:32.888460 20805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:32.888634 20805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:32.891024 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:32.869311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37641 (local address 127.0.0.1:39249)
0619 13:18:32.870057 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:32.870068 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:32.870107 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:18:32.871848 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:32.871870 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:32.872265 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:32.872569 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:32.874240 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:32.875886 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:32.877709 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:32.878472 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:32.878602 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:32.883500 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:32.883508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:32.883523 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:32.883566 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:32.886555 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:32.887498 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:32.887504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:32.887511 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:32.887732 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:32.888242 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:32.888246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:32.888249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:32.888868 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:32.889104 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:32.889620 (+ 516us) server_negotiation.cc:299] Negotiation successful
0619 13:18:32.889986 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":325,"threads_started":1}
I20260619 13:18:32.891247 20805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c53542cd1254cbeb6926beaa3e3ce2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39249 } }
I20260619 13:18:32.891773 20805 leader_election.cc:302] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c53542cd1254cbeb6926beaa3e3ce2c; no voters:
I20260619 13:18:32.892371 20805 leader_election.cc:288] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:32.892490 20810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:32.894078 20810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [term 1 LEADER]: Becoming Leader. State: Replica: 1c53542cd1254cbeb6926beaa3e3ce2c, State: Running, Role: LEADER
I20260619 13:18:32.894600 20805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:32.894686 20810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c53542cd1254cbeb6926beaa3e3ce2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39249 } }
I20260619 13:18:32.894958 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:32.897375 20812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c53542cd1254cbeb6926beaa3e3ce2c. Latest consensus state: current_term: 1 leader_uuid: "1c53542cd1254cbeb6926beaa3e3ce2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c53542cd1254cbeb6926beaa3e3ce2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39249 } } }
I20260619 13:18:32.897758 20812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:32.898231 20811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c53542cd1254cbeb6926beaa3e3ce2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c53542cd1254cbeb6926beaa3e3ce2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39249 } } }
I20260619 13:18:32.898516 20811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:32.901437 20819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:32.906946 20819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:32.930008 20819 catalog_manager.cc:1347] Generated new cluster ID: 4dd18a3a5f51414f8a0477a5472570b4
I20260619 13:18:32.930251 20819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:32.944169 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:32.918066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54265 (local address 127.0.0.1:39249)
0619 13:18:32.918237 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:32.918244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:32.926818 (+ 8574us) server_negotiation.cc:407] Connection header received
0619 13:18:32.927016 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:32.927021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:32.927108 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:32.927241 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:32.929294 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:32.930497 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:32.932374 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:32.933030 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:32.933127 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:32.936456 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:32.936463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:32.936468 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:32.936506 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:32.940180 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:32.941989 (+ 1809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:32.941995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:32.941999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:32.942146 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:32.942640 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:32.942645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:32.942648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:32.943016 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:32.943263 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:32.943656 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 13:18:32.943822 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:18:32.958166 20819 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:32.959995 20819 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:32.969929 20819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c53542cd1254cbeb6926beaa3e3ce2c: Generated new TSK 0
I20260619 13:18:32.971056 20819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:32.979471 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 20738
I20260619 13:18:32.980921 20800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:33.078414 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 20738
W20260619 13:18:33.092072 20723 connection.cc:535] client connection to 127.0.0.1:39249 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39249): Connection reset by peer (error 104)
2026-06-19T13:18:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1851 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:33 dist-test-slave-mkt3 krb5kdc[20837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:33 dist-test-slave-mkt3 krb5kdc[20837](info): set up 2 sockets
Jun 19 13:18:33 dist-test-slave-mkt3 krb5kdc[20837](info): commencing operation
krb5kdc: starting...
W20260619 13:18:34.081915 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:34 dist-test-slave-mkt3 krb5kdc[20837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:18:34.306375 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:34.617695 20850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:34.617971 20850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:34.618119 20850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:34.625056 20850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:34.625160 20850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:34.625237 20850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:34.625317 20850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:34.625377 20850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:34.634920 20850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44255
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:34.636119 20850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:34.637362 20850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:34.648247 20855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:34.648208 20856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:34.648168 20858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:34.649128 20850 server_base.cc:1034] running on GCE node
I20260619 13:18:34.649924 20850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:34.651286 20850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:34.652489 20850 hybrid_clock.cc:648] HybridClock initialized: now 1781875114652447 us; error 35 us; skew 500 ppm
I20260619 13:18:34.652822 20850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:34 dist-test-slave-mkt3 krb5kdc[20837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:34.655829 20850 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:18:34.657157 20850 webserver.cc:460] Webserver started at http://127.0.0.1:45853/ using document root <none> and password file <none>
I20260619 13:18:34.657768 20850 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:34.657896 20850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:34.658159 20850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:34.660940 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "0259536b320d473b9d6173df9a94ee48"
format_stamp: "Formatted at 2026-06-19 13:18:34 on dist-test-slave-mkt3"
server_key: "20fc97b1e80cc3e9d62869b77f12aeb9"
server_key_iv: "210c99234de5878e3f55808bb66100e7"
server_key_version: "encryptionkey@0"
I20260619 13:18:34.661653 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "0259536b320d473b9d6173df9a94ee48"
format_stamp: "Formatted at 2026-06-19 13:18:34 on dist-test-slave-mkt3"
server_key: "20fc97b1e80cc3e9d62869b77f12aeb9"
server_key_iv: "210c99234de5878e3f55808bb66100e7"
server_key_version: "encryptionkey@0"
I20260619 13:18:34.666527 20850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 13:18:34.669981 20865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:34.671343 20850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:18:34.671507 20850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "0259536b320d473b9d6173df9a94ee48"
format_stamp: "Formatted at 2026-06-19 13:18:34 on dist-test-slave-mkt3"
server_key: "20fc97b1e80cc3e9d62869b77f12aeb9"
server_key_iv: "210c99234de5878e3f55808bb66100e7"
server_key_version: "encryptionkey@0"
I20260619 13:18:34.671697 20850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:34.695708 20850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:34.699597 20850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:34.699844 20850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:34.716189 20850 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44255
I20260619 13:18:34.716218 20916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44255 every 8 connection(s)
I20260619 13:18:34.717475 20850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:34.721134 20917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:34.726759 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 20850
I20260619 13:18:34.726928 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:34.727344 388 external_mini_cluster.cc:1428] Setting key 0ad6bd9bc226e9c3fc02439d55388493
I20260619 13:18:34.731402 20917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48: Bootstrap starting.
I20260619 13:18:34.734615 20917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:34.735816 20917 log.cc:824] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:34.738952 20917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48: No bootstrap required, opened a new log
I20260619 13:18:34.740731 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:34.729236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53261 (local address 127.0.0.1:44255)
0619 13:18:34.730007 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:34.730018 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:34.730063 (+ 45us) server_negotiation.cc:407] Connection header received
0619 13:18:34.731699 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:34.731722 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:34.732105 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:34.732411 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:34.733014 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:34.734340 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:34.736140 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:34.737171 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:34.737297 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:34.738554 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:34.738562 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:34.738575 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:34.739119 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:34.739192 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:34.739216 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:34.739345 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:34.739614 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 13:18:34.739785 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":342,"threads_started":1}
I20260619 13:18:34.744226 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:34.746500 20917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0259536b320d473b9d6173df9a94ee48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44255 } }
I20260619 13:18:34.746918 20917 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:34.747008 20917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0259536b320d473b9d6173df9a94ee48, State: Initialized, Role: FOLLOWER
I20260619 13:18:34.747577 20917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0259536b320d473b9d6173df9a94ee48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44255 } }
I20260619 13:18:34.747797 20917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:34.747918 20917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:34.748081 20917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:34.750907 20917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0259536b320d473b9d6173df9a94ee48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44255 } }
I20260619 13:18:34.751473 20917 leader_election.cc:302] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0259536b320d473b9d6173df9a94ee48; no voters:
I20260619 13:18:34.752094 20917 leader_election.cc:288] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:34.752180 20927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:34.753479 20927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [term 1 LEADER]: Becoming Leader. State: Replica: 0259536b320d473b9d6173df9a94ee48, State: Running, Role: LEADER
I20260619 13:18:34.754101 20927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0259536b320d473b9d6173df9a94ee48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44255 } }
I20260619 13:18:34.754346 20917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:34.755339 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:34.748222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48467 (local address 127.0.0.1:44255)
0619 13:18:34.748397 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:34.748403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:34.748915 (+ 512us) server_negotiation.cc:407] Connection header received
0619 13:18:34.749085 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:34.749090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:34.749144 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:34.749222 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:34.749748 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:34.750493 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:34.752321 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:34.753031 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:34.753127 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:34.754496 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:34.754502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:34.754506 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:34.754527 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:34.754597 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:34.754619 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:34.754731 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:34.754959 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 13:18:34.755010 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:18:34.759449 20928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0259536b320d473b9d6173df9a94ee48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0259536b320d473b9d6173df9a94ee48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44255 } } }
I20260619 13:18:34.759835 20928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:34.760272 20929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0259536b320d473b9d6173df9a94ee48. Latest consensus state: current_term: 1 leader_uuid: "0259536b320d473b9d6173df9a94ee48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0259536b320d473b9d6173df9a94ee48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44255 } } }
I20260619 13:18:34.760643 20929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:34.762323 20937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:34.766283 20937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:34.773705 20937 catalog_manager.cc:1347] Generated new cluster ID: d54fecfebbec4b02a332375cb63aa841
I20260619 13:18:34.773813 20937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:34.798362 20937 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:34.799842 20937 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:34.813782 20937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0259536b320d473b9d6173df9a94ee48: Generated new TSK 0
I20260619 13:18:34.814615 20937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:34.830490 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 20850
W20260619 13:18:34.831548 20912 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53261
W20260619 13:18:34.832227 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 20850: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:18:34.832405 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 20850
W20260619 13:18:34.845954 20835 connection.cc:535] client connection to 127.0.0.1:44255 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44255): Connection reset by peer (error 104)
2026-06-19T13:18:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1753 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:34 dist-test-slave-mkt3 krb5kdc[20948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:34 dist-test-slave-mkt3 krb5kdc[20948](info): set up 2 sockets
Jun 19 13:18:34 dist-test-slave-mkt3 krb5kdc[20948](info): commencing operation
krb5kdc: starting...
W20260619 13:18:35.841893 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:35 dist-test-slave-mkt3 krb5kdc[20948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:18:36.062899 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34289
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:36.374593 20961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:36.374864 20961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:36.374986 20961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:36.381976 20961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:36.382076 20961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:36.382148 20961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:36.382221 20961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:36.382274 20961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:36.391978 20961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34289
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:36.393221 20961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:36.394483 20961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:36.405200 20966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:36.405479 20967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:36.405262 20969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:36.406080 20961 server_base.cc:1034] running on GCE node
I20260619 13:18:36.406744 20961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:36.408010 20961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:36.409226 20961 hybrid_clock.cc:648] HybridClock initialized: now 1781875116409176 us; error 39 us; skew 500 ppm
I20260619 13:18:36.409555 20961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:36 dist-test-slave-mkt3 krb5kdc[20948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:36.412523 20961 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:18:36.413949 20961 webserver.cc:460] Webserver started at http://127.0.0.1:36679/ using document root <none> and password file <none>
I20260619 13:18:36.414618 20961 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:36.414721 20961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:36.415004 20961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:36.417873 20961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "e39cb32513c143aababfd1b628fd1bfd"
format_stamp: "Formatted at 2026-06-19 13:18:36 on dist-test-slave-mkt3"
server_key: "738ed14feaec9ae69bac2756cd29ff90"
server_key_iv: "dbbee068dce69ce50d190f2af2bf150a"
server_key_version: "encryptionkey@0"
I20260619 13:18:36.418606 20961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "e39cb32513c143aababfd1b628fd1bfd"
format_stamp: "Formatted at 2026-06-19 13:18:36 on dist-test-slave-mkt3"
server_key: "738ed14feaec9ae69bac2756cd29ff90"
server_key_iv: "dbbee068dce69ce50d190f2af2bf150a"
server_key_version: "encryptionkey@0"
I20260619 13:18:36.423460 20961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:18:36.426891 20976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:36.428172 20961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 13:18:36.428316 20961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "e39cb32513c143aababfd1b628fd1bfd"
format_stamp: "Formatted at 2026-06-19 13:18:36 on dist-test-slave-mkt3"
server_key: "738ed14feaec9ae69bac2756cd29ff90"
server_key_iv: "dbbee068dce69ce50d190f2af2bf150a"
server_key_version: "encryptionkey@0"
I20260619 13:18:36.428495 20961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:36.434046 20961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:36.438009 20961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:36.438271 20961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:36.454733 20961 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34289
I20260619 13:18:36.454782 21027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34289 every 8 connection(s)
I20260619 13:18:36.456063 20961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:36.459647 21028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:36.464510 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 20961
I20260619 13:18:36.464672 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:36.465153 388 external_mini_cluster.cc:1428] Setting key 59a4fb65c0c6b0ccb1860d7ce703d5ba
I20260619 13:18:36.470284 21028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd: Bootstrap starting.
I20260619 13:18:36.472429 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:36.466838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52315 (local address 127.0.0.1:34289)
0619 13:18:36.467594 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:36.467605 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:36.467653 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:18:36.469326 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:36.469347 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:36.469665 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:36.470013 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:36.470995 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:36.471001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:36.471013 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:36.471026 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:36.471081 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:36.471101 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:36.471259 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:36.471376 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 13:18:36.471532 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":355,"threads_started":1}
I20260619 13:18:36.473462 21028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:36.474704 21028 log.cc:824] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:36.475878 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:36.477249 21028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd: No bootstrap required, opened a new log
I20260619 13:18:36.482066 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:36.479842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44013 (local address 127.0.0.1:34289)
0619 13:18:36.480007 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:36.480012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:36.480295 (+ 283us) server_negotiation.cc:407] Connection header received
0619 13:18:36.480454 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:36.480458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:36.480503 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:36.480574 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:36.481441 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:36.481446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:36.481452 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:36.481466 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:36.481502 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:36.481516 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:36.481603 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:36.481732 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 13:18:36.481782 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:18:36.485419 21028 raft_consensus.cc:348] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39cb32513c143aababfd1b628fd1bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34289 } }
I20260619 13:18:36.485762 21028 raft_consensus.cc:374] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:36.485869 21028 raft_consensus.cc:729] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39cb32513c143aababfd1b628fd1bfd, State: Initialized, Role: FOLLOWER
I20260619 13:18:36.486388 21028 consensus_queue.cc:260] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39cb32513c143aababfd1b628fd1bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34289 } }
I20260619 13:18:36.486617 21028 raft_consensus.cc:388] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:36.486738 21028 raft_consensus.cc:482] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:36.486886 21028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:36.489082 21028 raft_consensus.cc:504] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39cb32513c143aababfd1b628fd1bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34289 } }
I20260619 13:18:36.489631 21028 leader_election.cc:302] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e39cb32513c143aababfd1b628fd1bfd; no voters:
I20260619 13:18:36.490154 21028 leader_election.cc:288] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:36.490268 21038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:36.491109 21038 raft_consensus.cc:686] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [term 1 LEADER]: Becoming Leader. State: Replica: e39cb32513c143aababfd1b628fd1bfd, State: Running, Role: LEADER
I20260619 13:18:36.491573 21038 consensus_queue.cc:237] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39cb32513c143aababfd1b628fd1bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34289 } }
I20260619 13:18:36.491932 21028 sys_catalog.cc:564] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:36.494176 21039 sys_catalog.cc:455] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e39cb32513c143aababfd1b628fd1bfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39cb32513c143aababfd1b628fd1bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34289 } } }
I20260619 13:18:36.494472 21039 sys_catalog.cc:458] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:36.494745 21040 sys_catalog.cc:455] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e39cb32513c143aababfd1b628fd1bfd. Latest consensus state: current_term: 1 leader_uuid: "e39cb32513c143aababfd1b628fd1bfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e39cb32513c143aababfd1b628fd1bfd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34289 } } }
I20260619 13:18:36.495031 21040 sys_catalog.cc:458] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:36.495244 21045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:36.500573 21045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:36.509270 21045 catalog_manager.cc:1347] Generated new cluster ID: 2bdb0aa4139245dfb2d68c975ce25b4a
I20260619 13:18:36.509400 21045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:36.525938 21045 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:36.526475 21045 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:36.533468 21045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e39cb32513c143aababfd1b628fd1bfd: Generated new TSK 0
I20260619 13:18:36.534483 21045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:36.549342 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 20961
W20260619 13:18:36.550439 21023 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52315
W20260619 13:18:36.551203 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 20961: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:18:36.551375 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 20961
2026-06-19T13:18:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1720 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:36 dist-test-slave-mkt3 krb5kdc[21059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:36 dist-test-slave-mkt3 krb5kdc[21059](info): set up 2 sockets
Jun 19 13:18:36 dist-test-slave-mkt3 krb5kdc[21059](info): commencing operation
krb5kdc: starting...
W20260619 13:18:37.553709 388 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:37 dist-test-slave-mkt3 krb5kdc[21059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:18:37.774588 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:38.091926 21072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:38.092233 21072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:38.092355 21072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:38.099486 21072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:38.099593 21072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:38.099663 21072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:38.099748 21072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:38.099810 21072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:38.110287 21072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:38.111646 21072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:38.112921 21072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:38.124142 21080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:38.124285 21078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:38.124493 21077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:38.124971 21072 server_base.cc:1034] running on GCE node
I20260619 13:18:38.125790 21072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:38.127238 21072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:38.128437 21072 hybrid_clock.cc:648] HybridClock initialized: now 1781875118128382 us; error 47 us; skew 500 ppm
I20260619 13:18:38.128788 21072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:38 dist-test-slave-mkt3 krb5kdc[21059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:38.131814 21072 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:18:38.133203 21072 webserver.cc:460] Webserver started at http://127.0.0.1:45711/ using document root <none> and password file <none>
I20260619 13:18:38.133896 21072 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:38.134008 21072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:38.134297 21072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:38.137287 21072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "c901a7fb7e924508a2de668957342ea4"
format_stamp: "Formatted at 2026-06-19 13:18:38 on dist-test-slave-mkt3"
server_key: "90d72ce0e2b7d807deb08ce2fc416c50"
server_key_iv: "51a60f6ea9be4853ade621d479b3ed7c"
server_key_version: "encryptionkey@0"
I20260619 13:18:38.138056 21072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "c901a7fb7e924508a2de668957342ea4"
format_stamp: "Formatted at 2026-06-19 13:18:38 on dist-test-slave-mkt3"
server_key: "90d72ce0e2b7d807deb08ce2fc416c50"
server_key_iv: "51a60f6ea9be4853ade621d479b3ed7c"
server_key_version: "encryptionkey@0"
I20260619 13:18:38.142963 21072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:18:38.146181 21087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:38.147449 21072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:18:38.147631 21072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "c901a7fb7e924508a2de668957342ea4"
format_stamp: "Formatted at 2026-06-19 13:18:38 on dist-test-slave-mkt3"
server_key: "90d72ce0e2b7d807deb08ce2fc416c50"
server_key_iv: "51a60f6ea9be4853ade621d479b3ed7c"
server_key_version: "encryptionkey@0"
I20260619 13:18:38.147814 21072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:38.168156 21072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:38.172403 21072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:38.172683 21072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:38.188930 21072 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39121
I20260619 13:18:38.188952 21138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39121 every 8 connection(s)
I20260619 13:18:38.190269 21072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:38.193862 21139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:38.196215 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 21072
I20260619 13:18:38.196384 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:38.196878 388 external_mini_cluster.cc:1428] Setting key bafd06cac89df22df49aa6c8d66b467a
I20260619 13:18:38.205530 21139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4: Bootstrap starting.
I20260619 13:18:38.209131 21139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:38.210434 21139 log.cc:824] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:18:38 dist-test-slave-mkt3 krb5kdc[21059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 13:18:38.213236 21139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4: No bootstrap required, opened a new log
I20260619 13:18:38.218921 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:38.198598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54019 (local address 127.0.0.1:39121)
0619 13:18:38.199340 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:38.199354 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:38.199395 (+ 41us) server_negotiation.cc:407] Connection header received
0619 13:18:38.201029 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:38.201051 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:38.201442 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:38.201763 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:38.203581 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:38.204903 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:38.206619 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:38.207310 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:38.207443 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:38.212101 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:38.212110 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:38.212125 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:38.212159 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:38.215049 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:38.215709 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:38.215715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:38.215721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:38.215877 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:38.216247 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:38.216250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:38.216253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:38.216759 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:38.217060 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:38.217503 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 13:18:38.217930 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":330,"threads_started":1}
I20260619 13:18:38.220985 21139 raft_consensus.cc:348] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c901a7fb7e924508a2de668957342ea4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260619 13:18:38.221415 21139 raft_consensus.cc:374] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:38.221515 21139 raft_consensus.cc:729] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c901a7fb7e924508a2de668957342ea4, State: Initialized, Role: FOLLOWER
I20260619 13:18:38.222136 21139 consensus_queue.cc:260] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c901a7fb7e924508a2de668957342ea4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260619 13:18:38.222422 21139 raft_consensus.cc:388] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:38.222555 21139 raft_consensus.cc:482] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:38.222636 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:38.222728 21139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:38.225682 21139 raft_consensus.cc:504] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c901a7fb7e924508a2de668957342ea4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260619 13:18:38.226325 21139 leader_election.cc:302] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c901a7fb7e924508a2de668957342ea4; no voters:
I20260619 13:18:38.226938 21139 leader_election.cc:288] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:38.227021 21148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:38.229143 21139 sys_catalog.cc:564] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:38.233673 21148 raft_consensus.cc:686] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [term 1 LEADER]: Becoming Leader. State: Replica: c901a7fb7e924508a2de668957342ea4, State: Running, Role: LEADER
I20260619 13:18:38.234447 21148 consensus_queue.cc:237] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c901a7fb7e924508a2de668957342ea4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260619 13:18:38.237754 21149 sys_catalog.cc:455] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c901a7fb7e924508a2de668957342ea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c901a7fb7e924508a2de668957342ea4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } } }
I20260619 13:18:38.238168 21149 sys_catalog.cc:458] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:38.238706 21150 sys_catalog.cc:455] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c901a7fb7e924508a2de668957342ea4. Latest consensus state: current_term: 1 leader_uuid: "c901a7fb7e924508a2de668957342ea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c901a7fb7e924508a2de668957342ea4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } } }
I20260619 13:18:38.238968 21150 sys_catalog.cc:458] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:38.250527 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:38.228221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45889 (local address 127.0.0.1:39121)
0619 13:18:38.228417 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:38.228424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:38.233086 (+ 4662us) server_negotiation.cc:407] Connection header received
0619 13:18:38.233379 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:38.233386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:38.233482 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:38.233559 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:38.235626 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:38.237077 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:38.238968 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:38.239632 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:38.239756 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:38.242948 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:38.242956 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:38.242961 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:38.243011 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:38.246666 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:38.247505 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:38.247512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:38.247516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:38.247653 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:38.248074 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:38.248079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:38.248083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:38.248492 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:38.248720 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:38.249924 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0619 13:18:38.250164 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 13:18:38.254923 21162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:38.259294 21162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:38.267911 21162 catalog_manager.cc:1347] Generated new cluster ID: c8a8c58079af4443b5b347118ac86c67
I20260619 13:18:38.268043 21162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:38.275825 21162 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:38.277705 21162 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:38.301574 21162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c901a7fb7e924508a2de668957342ea4: Generated new TSK 0
I20260619 13:18:38.302743 21162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:38.319449 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 21072
I20260619 13:18:38.320921 21134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:38.412634 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 21072
2026-06-19T13:18:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1859 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:38 dist-test-slave-mkt3 krb5kdc[21172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:38 dist-test-slave-mkt3 krb5kdc[21172](info): set up 2 sockets
Jun 19 13:18:38 dist-test-slave-mkt3 krb5kdc[21172](info): commencing operation
krb5kdc: starting...
W20260619 13:18:39.421885 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:39 dist-test-slave-mkt3 krb5kdc[21172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:18:39.636461 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39601
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:39.945513 21185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:39.945787 21185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:39.945935 21185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:39.952880 21185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:39.952992 21185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:39.953069 21185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:39.953157 21185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:39.953222 21185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:39.962489 21185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:39.963663 21185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:39.964895 21185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:39.975726 21190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:39.975749 21191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:39.975730 21193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:39.976827 21185 server_base.cc:1034] running on GCE node
I20260619 13:18:39.977494 21185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:39.978816 21185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:39.980001 21185 hybrid_clock.cc:648] HybridClock initialized: now 1781875119979960 us; error 34 us; skew 500 ppm
I20260619 13:18:39.980309 21185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:39 dist-test-slave-mkt3 krb5kdc[21172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:39.986230 21185 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:18:39.987506 21185 webserver.cc:460] Webserver started at http://127.0.0.1:45561/ using document root <none> and password file <none>
I20260619 13:18:39.988104 21185 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:39.988199 21185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:39.988443 21185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:39.991242 21185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "cfddaff21502481a8c912327c2404ea1"
format_stamp: "Formatted at 2026-06-19 13:18:39 on dist-test-slave-mkt3"
server_key: "29a76e5f126778c6cb36bb4b868b75be"
server_key_iv: "ce8c69e1208b9da645b2f17cc8159fa6"
server_key_version: "encryptionkey@0"
I20260619 13:18:39.991910 21185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "cfddaff21502481a8c912327c2404ea1"
format_stamp: "Formatted at 2026-06-19 13:18:39 on dist-test-slave-mkt3"
server_key: "29a76e5f126778c6cb36bb4b868b75be"
server_key_iv: "ce8c69e1208b9da645b2f17cc8159fa6"
server_key_version: "encryptionkey@0"
I20260619 13:18:39.996541 21185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:18:39.999706 21200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:40.000910 21185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 13:18:40.001061 21185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "cfddaff21502481a8c912327c2404ea1"
format_stamp: "Formatted at 2026-06-19 13:18:39 on dist-test-slave-mkt3"
server_key: "29a76e5f126778c6cb36bb4b868b75be"
server_key_iv: "ce8c69e1208b9da645b2f17cc8159fa6"
server_key_version: "encryptionkey@0"
I20260619 13:18:40.001225 21185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:40.027523 21185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:40.031363 21185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:40.031610 21185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:40.048347 21185 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39601
I20260619 13:18:40.048386 21251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39601 every 8 connection(s)
I20260619 13:18:40.049651 21185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:40.053247 21252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:40.056435 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 21185
I20260619 13:18:40.056612 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:40.057087 388 external_mini_cluster.cc:1428] Setting key 038d4475384d52ece11c9161aca15f94
I20260619 13:18:40.064721 21252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1: Bootstrap starting.
I20260619 13:18:40.068091 21252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:40.069217 21252 log.cc:824] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:40.069681 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:40.058721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58371 (local address 127.0.0.1:39601)
0619 13:18:40.059414 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:40.059426 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:40.059465 (+ 39us) server_negotiation.cc:407] Connection header received
0619 13:18:40.061079 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:40.061099 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:40.061497 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:40.061818 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:40.062393 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:40.063676 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:40.065434 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:40.066141 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:40.066298 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:40.067470 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:40.067478 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:40.067490 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:40.067995 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:40.068068 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:40.068098 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:40.068188 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:40.068458 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 13:18:40.068722 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":324,"threads_started":1}
I20260619 13:18:40.072207 21252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1: No bootstrap required, opened a new log
I20260619 13:18:40.073387 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:40.080197 21252 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfddaff21502481a8c912327c2404ea1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39601 } }
I20260619 13:18:40.080538 21252 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:40.080628 21252 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfddaff21502481a8c912327c2404ea1, State: Initialized, Role: FOLLOWER
I20260619 13:18:40.081172 21252 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfddaff21502481a8c912327c2404ea1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39601 } }
I20260619 13:18:40.081400 21252 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:40.081538 21252 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:40.081732 21252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:40.083101 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:40.077349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38727 (local address 127.0.0.1:39601)
0619 13:18:40.077516 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:40.077521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:40.077786 (+ 265us) server_negotiation.cc:407] Connection header received
0619 13:18:40.077942 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:40.077947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:40.078024 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:40.078135 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:40.078582 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:40.079623 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:40.080926 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:40.081354 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:40.081470 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:40.082476 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:40.082480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:40.082484 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:40.082502 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:40.082542 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:40.082556 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:40.082626 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:40.082747 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 13:18:40.082814 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:18:40.084362 21252 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfddaff21502481a8c912327c2404ea1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39601 } }
I20260619 13:18:40.084800 21252 leader_election.cc:302] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfddaff21502481a8c912327c2404ea1; no voters:
I20260619 13:18:40.085305 21252 leader_election.cc:288] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:40.085384 21262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:40.086053 21262 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [term 1 LEADER]: Becoming Leader. State: Replica: cfddaff21502481a8c912327c2404ea1, State: Running, Role: LEADER
I20260619 13:18:40.086525 21262 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfddaff21502481a8c912327c2404ea1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39601 } }
I20260619 13:18:40.087270 21252 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:40.089814 21263 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfddaff21502481a8c912327c2404ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfddaff21502481a8c912327c2404ea1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39601 } } }
I20260619 13:18:40.090407 21263 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:40.089360 21264 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfddaff21502481a8c912327c2404ea1. Latest consensus state: current_term: 1 leader_uuid: "cfddaff21502481a8c912327c2404ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfddaff21502481a8c912327c2404ea1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39601 } } }
I20260619 13:18:40.091435 21264 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:40.094738 21272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:40.098714 21272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:40.106653 21272 catalog_manager.cc:1347] Generated new cluster ID: 72a88cbc6f014f7492012a6ab152a3e3
I20260619 13:18:40.106758 21272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:40.123669 21272 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:40.125433 21272 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:40.136274 21272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfddaff21502481a8c912327c2404ea1: Generated new TSK 0
I20260619 13:18:40.137030 21272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:40.148972 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 21185
W20260619 13:18:40.150005 21247 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58371
W20260619 13:18:40.150657 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 21185: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:18:40.150782 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 21185
2026-06-19T13:18:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1737 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:40 dist-test-slave-mkt3 krb5kdc[21283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:40 dist-test-slave-mkt3 krb5kdc[21283](info): set up 2 sockets
Jun 19 13:18:40 dist-test-slave-mkt3 krb5kdc[21283](info): commencing operation
krb5kdc: starting...
W20260619 13:18:41.153757 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:41 dist-test-slave-mkt3 krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:18:41.376145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44117
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:41.689610 21296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:41.689908 21296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:41.690039 21296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:41.697091 21296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:41.697199 21296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:41.697263 21296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:41.697342 21296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:41.697398 21296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:41.706861 21296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:41.708055 21296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:41.709317 21296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:41.720255 21302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:41.720216 21304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:41.720295 21301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:41.721026 21296 server_base.cc:1034] running on GCE node
I20260619 13:18:41.721709 21296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:41.723006 21296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:41.724205 21296 hybrid_clock.cc:648] HybridClock initialized: now 1781875121724153 us; error 32 us; skew 500 ppm
I20260619 13:18:41.724510 21296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:41 dist-test-slave-mkt3 krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:41.727423 21296 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:18:41.728750 21296 webserver.cc:460] Webserver started at http://127.0.0.1:38097/ using document root <none> and password file <none>
I20260619 13:18:41.729393 21296 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:41.729506 21296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:41.729784 21296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:41.732635 21296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "028dc2fc622b4e508d7c90c6862448b1"
format_stamp: "Formatted at 2026-06-19 13:18:41 on dist-test-slave-mkt3"
server_key: "532dedc3fdaeffc4c4f0e5943c85506d"
server_key_iv: "44389e094ed3966338b9b1a09be982ee"
server_key_version: "encryptionkey@0"
I20260619 13:18:41.733361 21296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "028dc2fc622b4e508d7c90c6862448b1"
format_stamp: "Formatted at 2026-06-19 13:18:41 on dist-test-slave-mkt3"
server_key: "532dedc3fdaeffc4c4f0e5943c85506d"
server_key_iv: "44389e094ed3966338b9b1a09be982ee"
server_key_version: "encryptionkey@0"
I20260619 13:18:41.738217 21296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 13:18:41.741441 21311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:41.742671 21296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:18:41.742828 21296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "028dc2fc622b4e508d7c90c6862448b1"
format_stamp: "Formatted at 2026-06-19 13:18:41 on dist-test-slave-mkt3"
server_key: "532dedc3fdaeffc4c4f0e5943c85506d"
server_key_iv: "44389e094ed3966338b9b1a09be982ee"
server_key_version: "encryptionkey@0"
I20260619 13:18:41.743021 21296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:41.748276 21296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:41.751907 21296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:41.752146 21296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:41.768221 21296 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44117
I20260619 13:18:41.768266 21362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44117 every 8 connection(s)
I20260619 13:18:41.769440 21296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:41.772956 21363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:41.777102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 21296
I20260619 13:18:41.777269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:41.777657 388 external_mini_cluster.cc:1428] Setting key 7907c7e9d784d5eeeedacfbe16af7a47
I20260619 13:18:41.783169 21363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1: Bootstrap starting.
I20260619 13:18:41.784832 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:41.779216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60713 (local address 127.0.0.1:44117)
0619 13:18:41.779929 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:41.779940 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:41.779980 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:18:41.781745 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:41.781765 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:41.782143 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:41.782448 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:41.783418 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:41.783424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:41.783436 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:41.783450 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:41.783511 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:41.783530 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:41.783710 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:41.783798 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 13:18:41.783950 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":306,"threads_started":1}
I20260619 13:18:41.786504 21363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:41.787572 21363 log.cc:824] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:41.788082 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:41.790025 21363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1: No bootstrap required, opened a new log
I20260619 13:18:41.794260 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:41.791962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46067 (local address 127.0.0.1:44117)
0619 13:18:41.792097 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:41.792100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:41.792532 (+ 432us) server_negotiation.cc:407] Connection header received
0619 13:18:41.792693 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:41.792698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:41.792748 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:41.792830 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:41.793617 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:41.793622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:41.793627 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:41.793644 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:41.793689 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:41.793704 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:41.793806 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:41.793919 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 13:18:41.793983 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 13:18:41.797633 21363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028dc2fc622b4e508d7c90c6862448b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44117 } }
I20260619 13:18:41.798017 21363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:41.798138 21363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 028dc2fc622b4e508d7c90c6862448b1, State: Initialized, Role: FOLLOWER
I20260619 13:18:41.798722 21363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028dc2fc622b4e508d7c90c6862448b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44117 } }
I20260619 13:18:41.798933 21363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:41.799057 21363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:41.799199 21363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:41.801330 21363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028dc2fc622b4e508d7c90c6862448b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44117 } }
I20260619 13:18:41.801801 21363 leader_election.cc:302] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 028dc2fc622b4e508d7c90c6862448b1; no voters:
I20260619 13:18:41.802348 21363 leader_election.cc:288] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:41.802450 21373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:41.803050 21373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [term 1 LEADER]: Becoming Leader. State: Replica: 028dc2fc622b4e508d7c90c6862448b1, State: Running, Role: LEADER
I20260619 13:18:41.803547 21373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028dc2fc622b4e508d7c90c6862448b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44117 } }
I20260619 13:18:41.804185 21363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:41.806283 21374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "028dc2fc622b4e508d7c90c6862448b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028dc2fc622b4e508d7c90c6862448b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44117 } } }
I20260619 13:18:41.806308 21375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 028dc2fc622b4e508d7c90c6862448b1. Latest consensus state: current_term: 1 leader_uuid: "028dc2fc622b4e508d7c90c6862448b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "028dc2fc622b4e508d7c90c6862448b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44117 } } }
I20260619 13:18:41.806974 21374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:41.806975 21375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:41.818041 21382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:41.822976 21382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:41.831974 21382 catalog_manager.cc:1347] Generated new cluster ID: 8a7c4033517743ef9dd9069b18e485c0
I20260619 13:18:41.832094 21382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:41.879860 21382 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:41.880285 21382 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:41.889494 21382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 028dc2fc622b4e508d7c90c6862448b1: Generated new TSK 0
I20260619 13:18:41.890362 21382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:41.901813 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 21296
W20260619 13:18:41.903070 21358 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60713
W20260619 13:18:41.903700 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 21296: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:18:41.903831 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 21296
2026-06-19T13:18:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1753 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:41 dist-test-slave-mkt3 krb5kdc[21394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:41 dist-test-slave-mkt3 krb5kdc[21394](info): set up 2 sockets
Jun 19 13:18:41 dist-test-slave-mkt3 krb5kdc[21394](info): commencing operation
krb5kdc: starting...
W20260619 13:18:42.905908 388 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:43 dist-test-slave-mkt3 krb5kdc[21394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:18:43.130309 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:43.439018 21407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:43.439316 21407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:43.439446 21407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:43.446463 21407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:43.446676 21407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:43.446746 21407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:43.446811 21407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:43.446868 21407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:43.456275 21407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:43.457446 21407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:43.458735 21407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:43.469620 21413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:43.469595 21412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:43.469648 21415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:43.470417 21407 server_base.cc:1034] running on GCE node
I20260619 13:18:43.471087 21407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:43.472381 21407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:43.473570 21407 hybrid_clock.cc:648] HybridClock initialized: now 1781875123473537 us; error 34 us; skew 500 ppm
I20260619 13:18:43.473944 21407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:43 dist-test-slave-mkt3 krb5kdc[21394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:43.476752 21407 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:18:43.478158 21407 webserver.cc:460] Webserver started at http://127.0.0.1:34679/ using document root <none> and password file <none>
I20260619 13:18:43.478753 21407 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:43.478853 21407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:43.479087 21407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:43.481887 21407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "6b6b1bb403d84ae98e9c9252a5beaca8"
format_stamp: "Formatted at 2026-06-19 13:18:43 on dist-test-slave-mkt3"
server_key: "0a215ba7f5f95fd6c6fca0633d71cf1d"
server_key_iv: "d54cf4d29ba58c8b5f38aeec9d49ff21"
server_key_version: "encryptionkey@0"
I20260619 13:18:43.482589 21407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "6b6b1bb403d84ae98e9c9252a5beaca8"
format_stamp: "Formatted at 2026-06-19 13:18:43 on dist-test-slave-mkt3"
server_key: "0a215ba7f5f95fd6c6fca0633d71cf1d"
server_key_iv: "d54cf4d29ba58c8b5f38aeec9d49ff21"
server_key_version: "encryptionkey@0"
I20260619 13:18:43.487197 21407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 13:18:43.490432 21422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:43.491750 21407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 13:18:43.491885 21407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "6b6b1bb403d84ae98e9c9252a5beaca8"
format_stamp: "Formatted at 2026-06-19 13:18:43 on dist-test-slave-mkt3"
server_key: "0a215ba7f5f95fd6c6fca0633d71cf1d"
server_key_iv: "d54cf4d29ba58c8b5f38aeec9d49ff21"
server_key_version: "encryptionkey@0"
I20260619 13:18:43.492038 21407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:43.510042 21407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:43.513820 21407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:43.514081 21407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:43.530933 21407 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45263
I20260619 13:18:43.530968 21473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45263 every 8 connection(s)
I20260619 13:18:43.532228 21407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:43.535970 21474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:43.540848 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 21407
I20260619 13:18:43.541060 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:43.541570 388 external_mini_cluster.cc:1428] Setting key 200b718ddfd375fcecd68a49175be537
I20260619 13:18:43.547070 21474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8: Bootstrap starting.
I20260619 13:18:43.550889 21474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:43.552075 21474 log.cc:824] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:43.554795 21474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8: No bootstrap required, opened a new log
Jun 19 13:18:43 dist-test-slave-mkt3 krb5kdc[21394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 13:18:43.564962 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:43.543407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:52689 (local address 127.0.0.1:45263)
0619 13:18:43.544156 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:43.544168 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:43.544217 (+ 49us) server_negotiation.cc:407] Connection header received
0619 13:18:43.545972 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:43.545998 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:43.546402 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:43.546747 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:43.548478 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:43.549907 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:43.551631 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:43.552354 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:43.552484 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:43.556615 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:43.556627 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:43.556657 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:43.556713 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:43.560608 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:43.561408 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:43.561420 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:43.561428 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:43.561625 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:43.562060 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:43.562065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:43.562068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:43.562743 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:43.563063 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:43.563365 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 13:18:43.563699 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":321,"threads_started":1}
I20260619 13:18:43.564740 21474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6b1bb403d84ae98e9c9252a5beaca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260619 13:18:43.565142 21474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:43.565254 21474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b6b1bb403d84ae98e9c9252a5beaca8, State: Initialized, Role: FOLLOWER
I20260619 13:18:43.566047 21474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6b1bb403d84ae98e9c9252a5beaca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260619 13:18:43.566347 21474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:43.566489 21474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:43.566660 21474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:43.569075 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:43.569291 21474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6b1bb403d84ae98e9c9252a5beaca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260619 13:18:43.569757 21474 leader_election.cc:302] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b6b1bb403d84ae98e9c9252a5beaca8; no voters:
I20260619 13:18:43.570325 21474 leader_election.cc:288] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:43.570465 21479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:43.572206 21474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:43.577282 21479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [term 1 LEADER]: Becoming Leader. State: Replica: 6b6b1bb403d84ae98e9c9252a5beaca8, State: Running, Role: LEADER
I20260619 13:18:43.577984 21479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6b1bb403d84ae98e9c9252a5beaca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260619 13:18:43.581833 21480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b6b1bb403d84ae98e9c9252a5beaca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6b1bb403d84ae98e9c9252a5beaca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } } }
I20260619 13:18:43.582276 21480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:43.582769 21481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b6b1bb403d84ae98e9c9252a5beaca8. Latest consensus state: current_term: 1 leader_uuid: "6b6b1bb403d84ae98e9c9252a5beaca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6b1bb403d84ae98e9c9252a5beaca8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } } }
I20260619 13:18:43.583041 21481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8 [sys.catalog]: This master's current role is: LEADER
W20260619 13:18:43.597149 21498 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:18:43.597271 21498 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:18:43.597415 21499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:43.598093 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:43.580311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:60043 (local address 127.0.0.1:45263)
0619 13:18:43.580471 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:43.580477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:43.581054 (+ 577us) server_negotiation.cc:407] Connection header received
0619 13:18:43.581327 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:43.581332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:43.581426 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:43.581507 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:43.583474 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:43.584700 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:43.586685 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:43.587420 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:43.587518 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:43.591005 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:43.591013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:43.591018 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:43.591058 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:43.594768 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:43.595680 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:43.595685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:43.595688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:43.595821 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:43.596352 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:43.596357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:43.596361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:43.596811 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:43.597085 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:43.597506 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 13:18:43.597698 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 13:18:43.599380 21499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:43.607827 21499 catalog_manager.cc:1347] Generated new cluster ID: 88ecdf54db554fe887e0ff212ea95dba
I20260619 13:18:43.607930 21499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:43.621618 21499 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:43.622867 21499 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:43.635070 21499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b6b1bb403d84ae98e9c9252a5beaca8: Generated new TSK 0
I20260619 13:18:43.636201 21499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:43.642154 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 21407
I20260619 13:18:43.643178 21469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:43.731379 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 21407
W20260619 13:18:43.747087 21392 connection.cc:535] client connection to 127.0.0.1:45263 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45263): Connection reset by peer (error 104)
2026-06-19T13:18:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1829 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:43 dist-test-slave-mkt3 krb5kdc[21506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:43 dist-test-slave-mkt3 krb5kdc[21506](info): set up 2 sockets
Jun 19 13:18:43 dist-test-slave-mkt3 krb5kdc[21506](info): commencing operation
krb5kdc: starting...
W20260619 13:18:44.749931 388 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:44 dist-test-slave-mkt3 krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:18:44.981092 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:45.293133 21519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:45.293417 21519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:45.293558 21519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:45.300611 21519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:45.300720 21519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:45.300798 21519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:45.300879 21519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:45.300937 21519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:45.310467 21519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:45.311703 21519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:45.312954 21519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:45.324071 21524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:45.323993 21527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:45.324023 21525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:45.324968 21519 server_base.cc:1034] running on GCE node
I20260619 13:18:45.325711 21519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:45.327108 21519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:45.328305 21519 hybrid_clock.cc:648] HybridClock initialized: now 1781875125328261 us; error 36 us; skew 500 ppm
I20260619 13:18:45.328632 21519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:45 dist-test-slave-mkt3 krb5kdc[21506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:45.339385 21519 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:18:45.340759 21519 webserver.cc:460] Webserver started at http://127.0.0.1:44717/ using document root <none> and password file <none>
I20260619 13:18:45.341427 21519 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:45.341529 21519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:45.341807 21519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:45.344646 21519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "5913b166804c4e49aeb92a32782d7fcd"
format_stamp: "Formatted at 2026-06-19 13:18:45 on dist-test-slave-mkt3"
server_key: "da5ed19f74b396f1733bfdccf2b00f58"
server_key_iv: "6eff28d888d125af51635f903f2c9b6d"
server_key_version: "encryptionkey@0"
I20260619 13:18:45.345350 21519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "5913b166804c4e49aeb92a32782d7fcd"
format_stamp: "Formatted at 2026-06-19 13:18:45 on dist-test-slave-mkt3"
server_key: "da5ed19f74b396f1733bfdccf2b00f58"
server_key_iv: "6eff28d888d125af51635f903f2c9b6d"
server_key_version: "encryptionkey@0"
I20260619 13:18:45.350121 21519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260619 13:18:45.353236 21534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:45.354445 21519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 13:18:45.354594 21519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "5913b166804c4e49aeb92a32782d7fcd"
format_stamp: "Formatted at 2026-06-19 13:18:45 on dist-test-slave-mkt3"
server_key: "da5ed19f74b396f1733bfdccf2b00f58"
server_key_iv: "6eff28d888d125af51635f903f2c9b6d"
server_key_version: "encryptionkey@0"
I20260619 13:18:45.354768 21519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:45.373219 21519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:45.376850 21519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:45.377146 21519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:45.393405 21519 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37155
I20260619 13:18:45.393440 21585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37155 every 8 connection(s)
I20260619 13:18:45.394688 21519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:45.398120 21586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:45.402073 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 21519
I20260619 13:18:45.402248 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:45.402766 388 external_mini_cluster.cc:1428] Setting key f074fbb55e99bcdb5911d7e6d89a2572
I20260619 13:18:45.410401 21586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd: Bootstrap starting.
I20260619 13:18:45.413815 21586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:45.414945 21586 log.cc:824] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:45.415345 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:45.404378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:33301 (local address 127.0.0.1:37155)
0619 13:18:45.405037 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:45.405049 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:45.405086 (+ 37us) server_negotiation.cc:407] Connection header received
0619 13:18:45.406669 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:45.406690 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:45.407093 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:45.407424 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:45.407960 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:45.409245 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:45.411114 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:45.411762 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:45.411900 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:45.413162 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:45.413170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:45.413184 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:45.413688 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:45.413763 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:45.413790 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:45.413968 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:45.414201 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 13:18:45.414381 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":293,"threads_started":1}
I20260619 13:18:45.417681 21586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd: No bootstrap required, opened a new log
I20260619 13:18:45.418998 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:45.427140 21586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5913b166804c4e49aeb92a32782d7fcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37155 } }
I20260619 13:18:45.427577 21586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:45.427688 21586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5913b166804c4e49aeb92a32782d7fcd, State: Initialized, Role: FOLLOWER
I20260619 13:18:45.428269 21586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5913b166804c4e49aeb92a32782d7fcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37155 } }
I20260619 13:18:45.428555 21586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:45.428697 21586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:45.428874 21586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:45.430167 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:45.423877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:46679 (local address 127.0.0.1:37155)
0619 13:18:45.424049 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:45.424055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:45.424380 (+ 325us) server_negotiation.cc:407] Connection header received
0619 13:18:45.424541 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:45.424545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:45.424608 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:45.424686 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:45.425266 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:45.426085 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:45.427812 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:45.428239 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:45.428337 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:45.429459 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:45.429464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:45.429467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:45.429482 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:45.429539 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:45.429557 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:45.429665 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:45.429831 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 13:18:45.429910 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:18:45.431226 21586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5913b166804c4e49aeb92a32782d7fcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37155 } }
I20260619 13:18:45.431723 21586 leader_election.cc:302] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5913b166804c4e49aeb92a32782d7fcd; no voters:
I20260619 13:18:45.432255 21586 leader_election.cc:288] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:45.432355 21596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:45.432964 21596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [term 1 LEADER]: Becoming Leader. State: Replica: 5913b166804c4e49aeb92a32782d7fcd, State: Running, Role: LEADER
I20260619 13:18:45.433451 21596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5913b166804c4e49aeb92a32782d7fcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37155 } }
I20260619 13:18:45.434145 21586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:45.436092 21598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5913b166804c4e49aeb92a32782d7fcd. Latest consensus state: current_term: 1 leader_uuid: "5913b166804c4e49aeb92a32782d7fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5913b166804c4e49aeb92a32782d7fcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37155 } } }
I20260619 13:18:45.436440 21598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:45.436822 21597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5913b166804c4e49aeb92a32782d7fcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5913b166804c4e49aeb92a32782d7fcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37155 } } }
I20260619 13:18:45.437134 21597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:45.443110 21602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:45.448549 21602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:45.458078 21602 catalog_manager.cc:1347] Generated new cluster ID: 8f83b5832e5541ebb416369d2d2cc82a
I20260619 13:18:45.458196 21602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:45.476007 21602 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:45.477751 21602 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:45.490053 21602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5913b166804c4e49aeb92a32782d7fcd: Generated new TSK 0
I20260619 13:18:45.491168 21602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:45.510984 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 21519
W20260619 13:18:45.512310 21581 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.141:33301
W20260619 13:18:45.513062 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 21519: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:18:45.513208 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 21519
W20260619 13:18:45.526636 21504 connection.cc:535] client connection to 127.0.0.1:37155 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37155): Connection reset by peer (error 104)
2026-06-19T13:18:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1779 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:45 dist-test-slave-mkt3 krb5kdc[21617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:45 dist-test-slave-mkt3 krb5kdc[21617](info): set up 2 sockets
Jun 19 13:18:45 dist-test-slave-mkt3 krb5kdc[21617](info): commencing operation
krb5kdc: starting...
W20260619 13:18:46.521947 388 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:46 dist-test-slave-mkt3 krb5kdc[21617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:18:46.756081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39623
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:47.072661 21630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:47.072943 21630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:47.073061 21630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:47.080523 21630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:47.080660 21630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:47.080737 21630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:47.080819 21630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:47.080878 21630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:47.090603 21630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:47.091840 21630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:47.093113 21630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:47.104314 21635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:47.104338 21636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:47.104262 21638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:47.105088 21630 server_base.cc:1034] running on GCE node
I20260619 13:18:47.105866 21630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:47.107245 21630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:47.108448 21630 hybrid_clock.cc:648] HybridClock initialized: now 1781875127108394 us; error 43 us; skew 500 ppm
I20260619 13:18:47.108789 21630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:47 dist-test-slave-mkt3 krb5kdc[21617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:47.111702 21630 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:18:47.113077 21630 webserver.cc:460] Webserver started at http://127.0.0.1:38201/ using document root <none> and password file <none>
I20260619 13:18:47.113734 21630 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:47.113878 21630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:47.114168 21630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:47.117035 21630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "1519823058ce4260bb61f6fffb015e49"
format_stamp: "Formatted at 2026-06-19 13:18:47 on dist-test-slave-mkt3"
server_key: "6ab74460a23f93b4ddff21cd00e7fc39"
server_key_iv: "9444e9bb28cf285a909afe6f012be089"
server_key_version: "encryptionkey@0"
I20260619 13:18:47.117760 21630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "1519823058ce4260bb61f6fffb015e49"
format_stamp: "Formatted at 2026-06-19 13:18:47 on dist-test-slave-mkt3"
server_key: "6ab74460a23f93b4ddff21cd00e7fc39"
server_key_iv: "9444e9bb28cf285a909afe6f012be089"
server_key_version: "encryptionkey@0"
I20260619 13:18:47.122663 21630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:18:47.126124 21645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:47.127441 21630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:18:47.127605 21630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "1519823058ce4260bb61f6fffb015e49"
format_stamp: "Formatted at 2026-06-19 13:18:47 on dist-test-slave-mkt3"
server_key: "6ab74460a23f93b4ddff21cd00e7fc39"
server_key_iv: "9444e9bb28cf285a909afe6f012be089"
server_key_version: "encryptionkey@0"
I20260619 13:18:47.127789 21630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:47.133190 21630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:47.137019 21630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:47.137321 21630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:47.152947 21630 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39623
I20260619 13:18:47.152982 21696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39623 every 8 connection(s)
I20260619 13:18:47.154309 21630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:47.157954 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 21630
I20260619 13:18:47.158078 21697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:47.158105 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:47.158512 388 external_mini_cluster.cc:1428] Setting key 409d6e4a8815b99ef7d50be72acdd613
I20260619 13:18:47.165874 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:47.160190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:60425 (local address 127.0.0.1:39623)
0619 13:18:47.160984 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:47.161008 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:47.161052 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:18:47.162576 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:47.162598 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:47.162945 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:47.163271 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:47.164326 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:47.164333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:47.164346 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:47.164361 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:47.164439 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:47.164460 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:47.164635 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:47.164734 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 13:18:47.164934 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":335,"threads_started":1}
I20260619 13:18:47.169545 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:47.171142 21697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49: Bootstrap starting.
I20260619 13:18:47.174376 21697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:47.175359 21697 log.cc:824] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:47.176028 21700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:47.173690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:45551 (local address 127.0.0.1:39623)
0619 13:18:47.173896 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:47.173903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:47.174318 (+ 415us) server_negotiation.cc:407] Connection header received
0619 13:18:47.174473 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:47.174478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:47.174543 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:47.174627 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:47.175427 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:47.175433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:47.175438 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:47.175456 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:47.175498 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:47.175516 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:47.175622 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:47.175705 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 13:18:47.175766 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 13:18:47.177968 21697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49: No bootstrap required, opened a new log
I20260619 13:18:47.184859 21697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1519823058ce4260bb61f6fffb015e49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39623 } }
I20260619 13:18:47.185202 21697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:47.185297 21697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1519823058ce4260bb61f6fffb015e49, State: Initialized, Role: FOLLOWER
I20260619 13:18:47.185825 21697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1519823058ce4260bb61f6fffb015e49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39623 } }
I20260619 13:18:47.186089 21697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:47.186214 21697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:47.186354 21697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:47.188899 21697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1519823058ce4260bb61f6fffb015e49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39623 } }
I20260619 13:18:47.189361 21697 leader_election.cc:302] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1519823058ce4260bb61f6fffb015e49; no voters:
I20260619 13:18:47.189903 21697 leader_election.cc:288] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:47.190033 21707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:47.190613 21707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [term 1 LEADER]: Becoming Leader. State: Replica: 1519823058ce4260bb61f6fffb015e49, State: Running, Role: LEADER
I20260619 13:18:47.191143 21707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1519823058ce4260bb61f6fffb015e49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39623 } }
I20260619 13:18:47.191797 21697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:47.195370 21709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1519823058ce4260bb61f6fffb015e49. Latest consensus state: current_term: 1 leader_uuid: "1519823058ce4260bb61f6fffb015e49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1519823058ce4260bb61f6fffb015e49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39623 } } }
I20260619 13:18:47.195362 21708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1519823058ce4260bb61f6fffb015e49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1519823058ce4260bb61f6fffb015e49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39623 } } }
I20260619 13:18:47.195809 21709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:47.195863 21708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:47.198380 21716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:47.203447 21716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:47.211681 21716 catalog_manager.cc:1347] Generated new cluster ID: fdb3c1a4af1c431faff1ad1d9e4f8b2e
I20260619 13:18:47.211800 21716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:47.230563 21716 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:47.230997 21716 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:47.239140 21716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1519823058ce4260bb61f6fffb015e49: Generated new TSK 0
I20260619 13:18:47.239943 21716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:47.250742 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 21630
W20260619 13:18:47.251910 21692 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.141:60425
W20260619 13:18:47.252558 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 21630: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:18:47.252687 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 21630
2026-06-19T13:18:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1739 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:47 dist-test-slave-mkt3 krb5kdc[21728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:47 dist-test-slave-mkt3 krb5kdc[21728](info): set up 2 sockets
Jun 19 13:18:47 dist-test-slave-mkt3 krb5kdc[21728](info): commencing operation
krb5kdc: starting...
W20260619 13:18:48.257998 388 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:48 dist-test-slave-mkt3 krb5kdc[21728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:18:48.492209 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:48.802621 21741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:48.802899 21741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:48.803016 21741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:48.810184 21741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:48.810292 21741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:48.810349 21741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:48.810432 21741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:48.810489 21741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:48.820163 21741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:48.821425 21741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:48.822716 21741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:48.833827 21747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:48.834025 21749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:48.833890 21746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:48.834587 21741 server_base.cc:1034] running on GCE node
I20260619 13:18:48.835289 21741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:48.836709 21741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:48.837935 21741 hybrid_clock.cc:648] HybridClock initialized: now 1781875128837883 us; error 28 us; skew 500 ppm
I20260619 13:18:48.838394 21741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:48 dist-test-slave-mkt3 krb5kdc[21728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:48.841810 21741 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:18:48.843336 21741 webserver.cc:460] Webserver started at http://127.0.0.1:33793/ using document root <none> and password file <none>
I20260619 13:18:48.843979 21741 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:48.844074 21741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:48.844336 21741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:48.847311 21741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "5d52602d46ae49b1acee7cb873ee71d3"
format_stamp: "Formatted at 2026-06-19 13:18:48 on dist-test-slave-mkt3"
server_key: "9e3040717dfed7fe27c3ab7d0a66c1c9"
server_key_iv: "1d4cf210fe02f0e8b0a93f3cb75a3853"
server_key_version: "encryptionkey@0"
I20260619 13:18:48.848011 21741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "5d52602d46ae49b1acee7cb873ee71d3"
format_stamp: "Formatted at 2026-06-19 13:18:48 on dist-test-slave-mkt3"
server_key: "9e3040717dfed7fe27c3ab7d0a66c1c9"
server_key_iv: "1d4cf210fe02f0e8b0a93f3cb75a3853"
server_key_version: "encryptionkey@0"
I20260619 13:18:48.853081 21741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 13:18:48.856417 21756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:48.857764 21741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 13:18:48.857939 21741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "5d52602d46ae49b1acee7cb873ee71d3"
format_stamp: "Formatted at 2026-06-19 13:18:48 on dist-test-slave-mkt3"
server_key: "9e3040717dfed7fe27c3ab7d0a66c1c9"
server_key_iv: "1d4cf210fe02f0e8b0a93f3cb75a3853"
server_key_version: "encryptionkey@0"
I20260619 13:18:48.858115 21741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:48.881948 21741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:48.886210 21741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:48.886441 21741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:48.902001 21741 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39625
I20260619 13:18:48.902025 21807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39625 every 8 connection(s)
I20260619 13:18:48.903282 21741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:48.903714 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 21741
I20260619 13:18:48.903877 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:48.904417 388 external_mini_cluster.cc:1428] Setting key b41a6a5b57d4fdd40de98157204cebe3
I20260619 13:18:48.907397 21808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 13:18:48 dist-test-slave-mkt3 krb5kdc[21728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 13:18:48.919684 21808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3: Bootstrap starting.
I20260619 13:18:48.922933 21808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:48.923908 21808 log.cc:824] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:48.926293 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:48.906179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:48551 (local address 127.0.0.1:39625)
0619 13:18:48.906874 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:48.906885 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:48.906923 (+ 38us) server_negotiation.cc:407] Connection header received
0619 13:18:48.908550 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:48.908579 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:48.908950 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:48.909249 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:48.910928 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:48.912235 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:48.914044 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:48.914696 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:48.914825 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:48.919397 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:48.919405 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:48.919420 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:48.919455 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:48.922354 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:48.923089 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:48.923095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:48.923101 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:48.923258 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:48.923677 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:48.923681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:48.923683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:48.924189 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:48.924484 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:48.924990 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 13:18:48.925319 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":301,"threads_started":1}
I20260619 13:18:48.926651 21808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3: No bootstrap required, opened a new log
I20260619 13:18:48.929872 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:48.935703 21808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d52602d46ae49b1acee7cb873ee71d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39625 } }
I20260619 13:18:48.936163 21808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:48.936265 21808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d52602d46ae49b1acee7cb873ee71d3, State: Initialized, Role: FOLLOWER
I20260619 13:18:48.936848 21808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d52602d46ae49b1acee7cb873ee71d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39625 } }
I20260619 13:18:48.937043 21808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:48.937173 21808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:48.937342 21808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:48.939944 21808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d52602d46ae49b1acee7cb873ee71d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39625 } }
I20260619 13:18:48.940543 21808 leader_election.cc:302] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d52602d46ae49b1acee7cb873ee71d3; no voters:
I20260619 13:18:48.941156 21808 leader_election.cc:288] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:48.941265 21818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:48.943203 21808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:48.956008 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:48.934908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:56757 (local address 127.0.0.1:39625)
0619 13:18:48.935089 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:48.935095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:48.935548 (+ 453us) server_negotiation.cc:407] Connection header received
0619 13:18:48.935677 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:48.935683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:48.935767 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:48.935875 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:48.937292 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:48.938478 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:48.939681 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:48.940283 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:48.940407 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:48.943529 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:48.943535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:48.943539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:48.943582 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:48.950001 (+ 6419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:48.951536 (+ 1535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:48.951542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:48.951547 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:48.951815 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:48.952491 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:48.952498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:48.952502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:48.953088 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:48.954909 (+ 1821us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:48.955187 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 13:18:48.955482 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:18:48.958282 21818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [term 1 LEADER]: Becoming Leader. State: Replica: 5d52602d46ae49b1acee7cb873ee71d3, State: Running, Role: LEADER
I20260619 13:18:48.958887 21818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d52602d46ae49b1acee7cb873ee71d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39625 } }
I20260619 13:18:48.962129 21820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d52602d46ae49b1acee7cb873ee71d3. Latest consensus state: current_term: 1 leader_uuid: "5d52602d46ae49b1acee7cb873ee71d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d52602d46ae49b1acee7cb873ee71d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39625 } } }
I20260619 13:18:48.962016 21819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d52602d46ae49b1acee7cb873ee71d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d52602d46ae49b1acee7cb873ee71d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39625 } } }
I20260619 13:18:48.962430 21820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:48.962440 21819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:48.964464 21831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:48.969408 21831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:48.977279 21831 catalog_manager.cc:1347] Generated new cluster ID: 6001073411b448cb9e3ec20e92f996bf
I20260619 13:18:48.977372 21831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:49.004616 21831 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:49.005962 21831 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:49.017455 21831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d52602d46ae49b1acee7cb873ee71d3: Generated new TSK 0
I20260619 13:18:49.018419 21831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:49.039650 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 21741
I20260619 13:18:49.040834 21803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:49.134071 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 21741
W20260619 13:18:49.147863 21726 connection.cc:535] client connection to 127.0.0.1:39625 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39625): Connection reset by peer (error 104)
2026-06-19T13:18:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1881 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:49 dist-test-slave-mkt3 krb5kdc[21840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:49 dist-test-slave-mkt3 krb5kdc[21840](info): set up 2 sockets
Jun 19 13:18:49 dist-test-slave-mkt3 krb5kdc[21840](info): commencing operation
krb5kdc: starting...
W20260619 13:18:50.149999 388 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:50 dist-test-slave-mkt3 krb5kdc[21840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:18:50.376927 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:50.686470 21853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:50.686784 21853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:50.686913 21853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:50.694182 21853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:50.694294 21853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:50.694363 21853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:50.694442 21853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:50.694497 21853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:50.704206 21853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39167
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:50.705386 21853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:50.706629 21853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:50.717330 21858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:50.717361 21859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:50.717396 21861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:50.717998 21853 server_base.cc:1034] running on GCE node
I20260619 13:18:50.718763 21853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:50.720032 21853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:50.721225 21853 hybrid_clock.cc:648] HybridClock initialized: now 1781875130721178 us; error 41 us; skew 500 ppm
I20260619 13:18:50.721530 21853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:50 dist-test-slave-mkt3 krb5kdc[21840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:50.733108 21853 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:18:50.734534 21853 webserver.cc:460] Webserver started at http://127.0.0.1:37171/ using document root <none> and password file <none>
I20260619 13:18:50.735204 21853 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:50.735318 21853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:50.735597 21853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:50.738459 21853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "915bdec6eb484952bca8aa58206929ed"
format_stamp: "Formatted at 2026-06-19 13:18:50 on dist-test-slave-mkt3"
server_key: "0e36c6cba1ef15ede6808e55a07e706b"
server_key_iv: "d7c9bf82a4fbec13fbf09c866479f36e"
server_key_version: "encryptionkey@0"
I20260619 13:18:50.739167 21853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "915bdec6eb484952bca8aa58206929ed"
format_stamp: "Formatted at 2026-06-19 13:18:50 on dist-test-slave-mkt3"
server_key: "0e36c6cba1ef15ede6808e55a07e706b"
server_key_iv: "d7c9bf82a4fbec13fbf09c866479f36e"
server_key_version: "encryptionkey@0"
I20260619 13:18:50.744169 21853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 13:18:50.747761 21868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:50.749081 21853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:18:50.749243 21853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "915bdec6eb484952bca8aa58206929ed"
format_stamp: "Formatted at 2026-06-19 13:18:50 on dist-test-slave-mkt3"
server_key: "0e36c6cba1ef15ede6808e55a07e706b"
server_key_iv: "d7c9bf82a4fbec13fbf09c866479f36e"
server_key_version: "encryptionkey@0"
I20260619 13:18:50.749421 21853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:50.768126 21853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:50.772064 21853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:50.772296 21853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:50.788817 21853 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39167
I20260619 13:18:50.788854 21919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39167 every 8 connection(s)
I20260619 13:18:50.790104 21853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:50.793512 21920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:50.798403 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 21853
I20260619 13:18:50.798569 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:50.798969 388 external_mini_cluster.cc:1428] Setting key 241cece18bc53fc7ccaaa47f8a545a41
I20260619 13:18:50.803644 21920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed: Bootstrap starting.
I20260619 13:18:50.806536 21920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:50.807485 21920 log.cc:824] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:50.810356 21920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed: No bootstrap required, opened a new log
I20260619 13:18:50.811190 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:50.800648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:53649 (local address 127.0.0.1:39167)
0619 13:18:50.801384 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:50.801399 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:50.801443 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:18:50.803106 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:50.803128 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:50.803500 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:50.803841 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:50.804331 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:50.805624 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:50.806985 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:50.807663 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:50.807814 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:50.809037 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:50.809044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:50.809058 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:50.809560 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:50.809651 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:50.809677 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:50.809796 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:50.810076 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 13:18:50.810237 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":325,"threads_started":1}
I20260619 13:18:50.815011 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:50.818229 21920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "915bdec6eb484952bca8aa58206929ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } }
I20260619 13:18:50.818548 21920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:50.818652 21920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915bdec6eb484952bca8aa58206929ed, State: Initialized, Role: FOLLOWER
I20260619 13:18:50.819341 21920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "915bdec6eb484952bca8aa58206929ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } }
I20260619 13:18:50.819617 21920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:50.819751 21920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:50.819921 21920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:50.822129 21920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "915bdec6eb484952bca8aa58206929ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } }
I20260619 13:18:50.822584 21920 leader_election.cc:302] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 915bdec6eb484952bca8aa58206929ed; no voters:
I20260619 13:18:50.823197 21920 leader_election.cc:288] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:50.823297 21930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:50.824606 21930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [term 1 LEADER]: Becoming Leader. State: Replica: 915bdec6eb484952bca8aa58206929ed, State: Running, Role: LEADER
I20260619 13:18:50.825361 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:50.819175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:50175 (local address 127.0.0.1:39167)
0619 13:18:50.819336 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:50.819342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:50.819692 (+ 350us) server_negotiation.cc:407] Connection header received
0619 13:18:50.819871 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:50.819880 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:50.819947 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:50.820025 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:50.820513 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:50.821294 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:50.822716 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:50.823211 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:50.823319 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:50.824550 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:50.824557 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:50.824561 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:50.824586 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:50.824669 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:50.824699 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:50.824800 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:50.825021 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 13:18:50.825071 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 13:18:50.825435 21920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:50.825436 21930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "915bdec6eb484952bca8aa58206929ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } }
I20260619 13:18:50.829387 21932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 915bdec6eb484952bca8aa58206929ed. Latest consensus state: current_term: 1 leader_uuid: "915bdec6eb484952bca8aa58206929ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "915bdec6eb484952bca8aa58206929ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } } }
I20260619 13:18:50.829406 21931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "915bdec6eb484952bca8aa58206929ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "915bdec6eb484952bca8aa58206929ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } } }
I20260619 13:18:50.829811 21932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:50.829815 21931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:50.835446 21940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:50.840163 21940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:50.849478 21940 catalog_manager.cc:1347] Generated new cluster ID: 7e54a42a496f478b9da13a7ab0e86357
I20260619 13:18:50.849583 21940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:50.871170 21940 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:50.872356 21940 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:50.884989 21940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 915bdec6eb484952bca8aa58206929ed: Generated new TSK 0
I20260619 13:18:50.885833 21940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:50.900225 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 21853
W20260619 13:18:50.901350 21915 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.141:53649
W20260619 13:18:50.902129 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 21853: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:18:50.902274 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 21853
W20260619 13:18:50.915392 21838 connection.cc:535] client connection to 127.0.0.1:39167 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39167): Connection reset by peer (error 104)
2026-06-19T13:18:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1769 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:50 dist-test-slave-mkt3 krb5kdc[21951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:50 dist-test-slave-mkt3 krb5kdc[21951](info): set up 2 sockets
Jun 19 13:18:50 dist-test-slave-mkt3 krb5kdc[21951](info): commencing operation
krb5kdc: starting...
W20260619 13:18:51.905984 388 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:52 dist-test-slave-mkt3 krb5kdc[21951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 13:18:52.138352 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43697
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:52.453079 21964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:52.453374 21964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:52.453524 21964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:52.460896 21964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:52.461017 21964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:52.461093 21964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:52.461179 21964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:52.461243 21964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:52.471035 21964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43697
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:52.472294 21964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:52.473551 21964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:52.484596 21969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:52.484570 21970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:52.484875 21972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:52.485306 21964 server_base.cc:1034] running on GCE node
I20260619 13:18:52.486140 21964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:52.487514 21964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:52.488739 21964 hybrid_clock.cc:648] HybridClock initialized: now 1781875132488689 us; error 45 us; skew 500 ppm
I20260619 13:18:52.489105 21964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:52 dist-test-slave-mkt3 krb5kdc[21951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:52.492148 21964 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 13:18:52.493597 21964 webserver.cc:460] Webserver started at http://127.0.0.1:42297/ using document root <none> and password file <none>
I20260619 13:18:52.494305 21964 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:52.494429 21964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:52.494720 21964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:52.497542 21964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "1221418797d84968a2b5a66a9a2c698e"
format_stamp: "Formatted at 2026-06-19 13:18:52 on dist-test-slave-mkt3"
server_key: "836c55f824e5e3e9fa7a903b5f2f118d"
server_key_iv: "debed5757a19c7a9b53a405b1a545035"
server_key_version: "encryptionkey@0"
I20260619 13:18:52.498332 21964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "1221418797d84968a2b5a66a9a2c698e"
format_stamp: "Formatted at 2026-06-19 13:18:52 on dist-test-slave-mkt3"
server_key: "836c55f824e5e3e9fa7a903b5f2f118d"
server_key_iv: "debed5757a19c7a9b53a405b1a545035"
server_key_version: "encryptionkey@0"
I20260619 13:18:52.503434 21964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 13:18:52.507006 21979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:52.508455 21964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 13:18:52.508613 21964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "1221418797d84968a2b5a66a9a2c698e"
format_stamp: "Formatted at 2026-06-19 13:18:52 on dist-test-slave-mkt3"
server_key: "836c55f824e5e3e9fa7a903b5f2f118d"
server_key_iv: "debed5757a19c7a9b53a405b1a545035"
server_key_version: "encryptionkey@0"
I20260619 13:18:52.508836 21964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:52.514547 21964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:52.518477 21964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:52.518755 21964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:52.535599 21964 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43697
I20260619 13:18:52.535626 22030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43697 every 8 connection(s)
I20260619 13:18:52.537001 21964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:52.540288 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 21964
I20260619 13:18:52.540472 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:52.540982 388 external_mini_cluster.cc:1428] Setting key a9467fd20ecfc9c3d050ba1175053ba7
I20260619 13:18:52.541497 22031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:52.548991 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:52.542759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:59029 (local address 127.0.0.1:43697)
0619 13:18:52.543488 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:52.543504 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:52.543553 (+ 49us) server_negotiation.cc:407] Connection header received
0619 13:18:52.545172 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:52.545200 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:52.545648 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:52.546113 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:52.547066 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:52.547075 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:52.547091 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:52.547109 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:52.547209 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:52.547236 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:52.547457 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:52.547580 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 13:18:52.547783 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":277,"threads_started":1}
I20260619 13:18:52.552419 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:52.555207 22031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e: Bootstrap starting.
I20260619 13:18:52.558126 22031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:52.559230 22031 log.cc:824] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:52.561599 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:52.559179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.141:48719 (local address 127.0.0.1:43697)
0619 13:18:52.559363 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:52.559368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:52.559816 (+ 448us) server_negotiation.cc:407] Connection header received
0619 13:18:52.559994 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:52.559999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:52.560061 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:52.560173 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:52.561049 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:52.561053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:52.561056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:52.561070 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:52.561105 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:52.561120 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:52.561209 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:52.561304 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 13:18:52.561348 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 13:18:52.562505 22031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e: No bootstrap required, opened a new log
I20260619 13:18:52.569866 22031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1221418797d84968a2b5a66a9a2c698e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43697 } }
I20260619 13:18:52.570277 22031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:52.570372 22031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1221418797d84968a2b5a66a9a2c698e, State: Initialized, Role: FOLLOWER
I20260619 13:18:52.570924 22031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1221418797d84968a2b5a66a9a2c698e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43697 } }
I20260619 13:18:52.571128 22031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:52.571255 22031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:52.571399 22031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:52.573606 22031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1221418797d84968a2b5a66a9a2c698e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43697 } }
I20260619 13:18:52.574172 22031 leader_election.cc:302] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1221418797d84968a2b5a66a9a2c698e; no voters:
I20260619 13:18:52.574714 22031 leader_election.cc:288] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:52.574817 22041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:52.575415 22041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [term 1 LEADER]: Becoming Leader. State: Replica: 1221418797d84968a2b5a66a9a2c698e, State: Running, Role: LEADER
I20260619 13:18:52.575875 22041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1221418797d84968a2b5a66a9a2c698e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43697 } }
I20260619 13:18:52.576565 22031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:52.579052 22042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1221418797d84968a2b5a66a9a2c698e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1221418797d84968a2b5a66a9a2c698e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43697 } } }
I20260619 13:18:52.578794 22043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1221418797d84968a2b5a66a9a2c698e. Latest consensus state: current_term: 1 leader_uuid: "1221418797d84968a2b5a66a9a2c698e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1221418797d84968a2b5a66a9a2c698e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43697 } } }
I20260619 13:18:52.579448 22043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:52.581516 22042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:52.581791 22050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:52.587172 22050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:52.595969 22050 catalog_manager.cc:1347] Generated new cluster ID: 6edf595a3c4b442eb2268dc1e6c38c77
I20260619 13:18:52.596112 22050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:52.608536 22050 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:52.608949 22050 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:52.615451 22050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1221418797d84968a2b5a66a9a2c698e: Generated new TSK 0
I20260619 13:18:52.616237 22050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:52.623040 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 21964
W20260619 13:18:52.624456 22026 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.141:59029
W20260619 13:18:52.625124 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 21964: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:18:52.625260 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 21964
2026-06-19T13:18:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1720 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:52 dist-test-slave-mkt3 krb5kdc[22062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:52 dist-test-slave-mkt3 krb5kdc[22062](info): set up 2 sockets
Jun 19 13:18:52 dist-test-slave-mkt3 krb5kdc[22062](info): commencing operation
krb5kdc: starting...
W20260619 13:18:53.638002 388 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:53 dist-test-slave-mkt3 krb5kdc[22062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:18:53.860944 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45731
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:54.172183 22075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:54.172451 22075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:54.172580 22075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:54.179595 22075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:54.179693 22075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:54.179764 22075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:54.179840 22075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:54.179898 22075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:54.189404 22075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45731
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:54.190685 22075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:54.191941 22075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:54.202793 22080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:54.202826 22081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:54.202874 22083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:54.203895 22075 server_base.cc:1034] running on GCE node
I20260619 13:18:54.204587 22075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:54.205931 22075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:54.207181 22075 hybrid_clock.cc:648] HybridClock initialized: now 1781875134207114 us; error 51 us; skew 500 ppm
I20260619 13:18:54.207520 22075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:54 dist-test-slave-mkt3 krb5kdc[22062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:54.210500 22075 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:18:54.211809 22075 webserver.cc:460] Webserver started at http://127.0.97.62:39215/ using document root <none> and password file <none>
I20260619 13:18:54.212446 22075 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:54.212564 22075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:54.212837 22075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:54.215648 22075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "c2f98907883a465aaeebfc7e65162f63"
format_stamp: "Formatted at 2026-06-19 13:18:54 on dist-test-slave-mkt3"
server_key: "15b095de3884244dd4ac98579951699b"
server_key_iv: "9dd4e93b3c857e5b6e3d4fb6fc7961d5"
server_key_version: "encryptionkey@0"
I20260619 13:18:54.216359 22075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "c2f98907883a465aaeebfc7e65162f63"
format_stamp: "Formatted at 2026-06-19 13:18:54 on dist-test-slave-mkt3"
server_key: "15b095de3884244dd4ac98579951699b"
server_key_iv: "9dd4e93b3c857e5b6e3d4fb6fc7961d5"
server_key_version: "encryptionkey@0"
I20260619 13:18:54.221189 22075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 13:18:54.224438 22090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:54.225733 22075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 13:18:54.225932 22075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "c2f98907883a465aaeebfc7e65162f63"
format_stamp: "Formatted at 2026-06-19 13:18:54 on dist-test-slave-mkt3"
server_key: "15b095de3884244dd4ac98579951699b"
server_key_iv: "9dd4e93b3c857e5b6e3d4fb6fc7961d5"
server_key_version: "encryptionkey@0"
I20260619 13:18:54.226099 22075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:54.236742 22075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:54.240556 22075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:54.240803 22075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:54.257246 22075 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45731
I20260619 13:18:54.257268 22141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45731 every 8 connection(s)
I20260619 13:18:54.258616 22075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:54.261471 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 22075
I20260619 13:18:54.261626 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:54.262058 388 external_mini_cluster.cc:1428] Setting key 3f9abff412ae0e67fe86b27db37b43b1
I20260619 13:18:54.262385 22142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:54.275967 22142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63: Bootstrap starting.
Jun 19 13:18:54 dist-test-slave-mkt3 krb5kdc[22062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:18:54.279878 22142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:54.281296 22142 log.cc:824] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:54.284705 22142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63: No bootstrap required, opened a new log
I20260619 13:18:54.286175 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:54.263735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56641 (local address 127.0.97.62:45731)
0619 13:18:54.264476 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:54.264491 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:54.264539 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:18:54.266882 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:54.266910 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:54.267425 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:54.267856 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:54.269127 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:54.271052 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:54.272286 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:54.273346 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:54.273535 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:54.277522 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:54.277533 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:54.277553 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:54.277597 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:54.281628 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:54.282584 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:54.282590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:54.282596 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:54.282805 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:54.283198 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:54.283202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:54.283205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:54.283863 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:54.284176 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:54.284505 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 13:18:54.284829 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":316,"threads_started":1}
I20260619 13:18:54.289944 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:54.294262 22142 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f98907883a465aaeebfc7e65162f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45731 } }
I20260619 13:18:54.294751 22142 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:54.294869 22142 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f98907883a465aaeebfc7e65162f63, State: Initialized, Role: FOLLOWER
I20260619 13:18:54.295583 22142 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f98907883a465aaeebfc7e65162f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45731 } }
I20260619 13:18:54.295862 22142 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:54.296012 22142 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:54.296180 22142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:54.298388 22142 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f98907883a465aaeebfc7e65162f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45731 } }
I20260619 13:18:54.298858 22142 leader_election.cc:302] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2f98907883a465aaeebfc7e65162f63; no voters:
I20260619 13:18:54.299433 22142 leader_election.cc:288] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:54.299537 22152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:54.301807 22142 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:54.302119 22152 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [term 1 LEADER]: Becoming Leader. State: Replica: c2f98907883a465aaeebfc7e65162f63, State: Running, Role: LEADER
I20260619 13:18:54.302738 22152 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f98907883a465aaeebfc7e65162f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45731 } }
I20260619 13:18:54.306309 22153 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2f98907883a465aaeebfc7e65162f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f98907883a465aaeebfc7e65162f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45731 } } }
I20260619 13:18:54.307130 22153 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:54.306846 22154 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2f98907883a465aaeebfc7e65162f63. Latest consensus state: current_term: 1 leader_uuid: "c2f98907883a465aaeebfc7e65162f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f98907883a465aaeebfc7e65162f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45731 } } }
I20260619 13:18:54.307618 22154 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:54.310717 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:54.295108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55161 (local address 127.0.97.62:45731)
0619 13:18:54.295294 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:54.295306 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:54.295820 (+ 514us) server_negotiation.cc:407] Connection header received
0619 13:18:54.295993 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:54.295999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:54.296069 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:54.296186 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:54.297484 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:54.298293 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:54.299633 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:54.300297 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:54.300415 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:54.302933 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:54.302938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:54.302941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:54.302970 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:54.306367 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:54.307331 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:54.307336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:54.307340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:54.307490 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:54.308060 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:54.308066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:54.308069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:54.308505 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:54.309923 (+ 1418us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:54.310131 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 13:18:54.310331 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:18:54.313031 22160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:54.318781 22160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:54.328020 22160 catalog_manager.cc:1347] Generated new cluster ID: 6b3f9a8dcc0040bdabeb2ff9d5ad837c
I20260619 13:18:54.328116 22160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:54.339241 22160 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:54.340473 22160 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:54.347966 22160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2f98907883a465aaeebfc7e65162f63: Generated new TSK 0
I20260619 13:18:54.348706 22160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:54.353163 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 22075
I20260619 13:18:54.354185 22137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:54.442524 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 22075
W20260619 13:18:54.455931 22060 connection.cc:535] client connection to 127.0.97.62:45731 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:45731): Connection reset by peer (error 104)
2026-06-19T13:18:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1817 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:54 dist-test-slave-mkt3 krb5kdc[22174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:54 dist-test-slave-mkt3 krb5kdc[22174](info): set up 2 sockets
Jun 19 13:18:54 dist-test-slave-mkt3 krb5kdc[22174](info): commencing operation
krb5kdc: starting...
W20260619 13:18:55.450038 388 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.007s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:55 dist-test-slave-mkt3 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:18:55.668354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45227
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:55.980394 22187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:55.980682 22187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:55.980815 22187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:55.988255 22187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:55.988365 22187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:55.988437 22187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:55.988516 22187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:55.988575 22187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:55.998104 22187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45227
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:55.999320 22187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:56.000556 22187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:56.011816 22195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:56.011842 22193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:56.011866 22192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:56.012949 22187 server_base.cc:1034] running on GCE node
I20260619 13:18:56.013696 22187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:56.015169 22187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:56.016362 22187 hybrid_clock.cc:648] HybridClock initialized: now 1781875136016321 us; error 36 us; skew 500 ppm
I20260619 13:18:56.016685 22187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:56 dist-test-slave-mkt3 krb5kdc[22174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:56.028844 22187 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:18:56.030429 22187 webserver.cc:460] Webserver started at http://127.0.97.62:44169/ using document root <none> and password file <none>
I20260619 13:18:56.031107 22187 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:56.031234 22187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:56.031524 22187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:56.034476 22187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "b62bbce0901d426a93b0c2f6b4c91af8"
format_stamp: "Formatted at 2026-06-19 13:18:56 on dist-test-slave-mkt3"
server_key: "ad88ed34cdd34a765f7ac080fad39e77"
server_key_iv: "d1b1d800a23abc27cfaba0d950c3d43f"
server_key_version: "encryptionkey@0"
I20260619 13:18:56.035205 22187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "b62bbce0901d426a93b0c2f6b4c91af8"
format_stamp: "Formatted at 2026-06-19 13:18:56 on dist-test-slave-mkt3"
server_key: "ad88ed34cdd34a765f7ac080fad39e77"
server_key_iv: "d1b1d800a23abc27cfaba0d950c3d43f"
server_key_version: "encryptionkey@0"
I20260619 13:18:56.040299 22187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:18:56.043697 22202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:56.044942 22187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 13:18:56.045099 22187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "b62bbce0901d426a93b0c2f6b4c91af8"
format_stamp: "Formatted at 2026-06-19 13:18:56 on dist-test-slave-mkt3"
server_key: "ad88ed34cdd34a765f7ac080fad39e77"
server_key_iv: "d1b1d800a23abc27cfaba0d950c3d43f"
server_key_version: "encryptionkey@0"
I20260619 13:18:56.045284 22187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:56.062651 22187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:56.066296 22187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:56.066532 22187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:56.083324 22187 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45227
I20260619 13:18:56.083348 22253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45227 every 8 connection(s)
I20260619 13:18:56.084606 22187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:56.088361 22254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:56.089820 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 22187
I20260619 13:18:56.090015 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:56.090685 388 external_mini_cluster.cc:1428] Setting key 87a2c71ee7f9605c7550eaaad0f9b45d
I20260619 13:18:56.101002 22254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8: Bootstrap starting.
I20260619 13:18:56.103210 22257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:56.092537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58623 (local address 127.0.97.62:45227)
0619 13:18:56.093431 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:56.093445 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:56.093485 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:18:56.095081 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:56.095102 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:56.095482 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:56.095827 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:56.096328 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:56.097560 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:56.098945 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:56.099673 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:56.099836 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:56.101056 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:56.101064 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:56.101083 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:56.101622 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:56.101692 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:56.101716 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:56.101874 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:56.102149 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 13:18:56.102307 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":473,"threads_started":1}
I20260619 13:18:56.104100 22254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:56.105095 22254 log.cc:824] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:56.106788 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:56.108027 22254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8: No bootstrap required, opened a new log
I20260619 13:18:56.117395 22254 raft_consensus.cc:348] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62bbce0901d426a93b0c2f6b4c91af8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45227 } }
I20260619 13:18:56.117864 22257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:56.111736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34785 (local address 127.0.97.62:45227)
0619 13:18:56.111906 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:56.111912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:56.112211 (+ 299us) server_negotiation.cc:407] Connection header received
0619 13:18:56.112340 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:56.112345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:56.112414 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:56.112512 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:56.112959 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:56.114119 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:56.115363 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:56.116046 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:56.116166 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:56.117165 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:56.117172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:56.117176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:56.117197 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:56.117257 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:56.117278 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:56.117392 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:56.117490 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 13:18:56.117539 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 13:18:56.117888 22254 raft_consensus.cc:374] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:56.117996 22254 raft_consensus.cc:729] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b62bbce0901d426a93b0c2f6b4c91af8, State: Initialized, Role: FOLLOWER
I20260619 13:18:56.118625 22254 consensus_queue.cc:260] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62bbce0901d426a93b0c2f6b4c91af8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45227 } }
I20260619 13:18:56.118856 22254 raft_consensus.cc:388] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:56.118984 22254 raft_consensus.cc:482] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:56.119181 22254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:56.121217 22254 raft_consensus.cc:504] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62bbce0901d426a93b0c2f6b4c91af8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45227 } }
I20260619 13:18:56.121685 22254 leader_election.cc:302] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b62bbce0901d426a93b0c2f6b4c91af8; no voters:
I20260619 13:18:56.122215 22254 leader_election.cc:288] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:56.122318 22264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:56.122941 22264 raft_consensus.cc:686] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [term 1 LEADER]: Becoming Leader. State: Replica: b62bbce0901d426a93b0c2f6b4c91af8, State: Running, Role: LEADER
I20260619 13:18:56.123498 22264 consensus_queue.cc:237] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62bbce0901d426a93b0c2f6b4c91af8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45227 } }
I20260619 13:18:56.124013 22254 sys_catalog.cc:564] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:56.125891 22266 sys_catalog.cc:455] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b62bbce0901d426a93b0c2f6b4c91af8. Latest consensus state: current_term: 1 leader_uuid: "b62bbce0901d426a93b0c2f6b4c91af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62bbce0901d426a93b0c2f6b4c91af8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45227 } } }
I20260619 13:18:56.126192 22266 sys_catalog.cc:458] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:56.126834 22265 sys_catalog.cc:455] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b62bbce0901d426a93b0c2f6b4c91af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62bbce0901d426a93b0c2f6b4c91af8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45227 } } }
I20260619 13:18:56.127103 22265 sys_catalog.cc:458] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:56.127722 22269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:56.132795 22269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:56.141644 22269 catalog_manager.cc:1347] Generated new cluster ID: a85894c37ae44670b8b3a505ebc746e6
I20260619 13:18:56.141741 22269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:56.167917 22269 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:56.169553 22269 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:56.180614 22269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b62bbce0901d426a93b0c2f6b4c91af8: Generated new TSK 0
I20260619 13:18:56.181444 22269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:56.193989 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 22187
W20260619 13:18:56.195116 22249 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58623
W20260619 13:18:56.195775 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 22187: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:18:56.195907 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 22187
W20260619 13:18:56.209583 22172 connection.cc:535] client connection to 127.0.97.62:45227 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:45227): Connection reset by peer (error 104)
2026-06-19T13:18:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1753 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:56 dist-test-slave-mkt3 krb5kdc[22285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:56 dist-test-slave-mkt3 krb5kdc[22285](info): set up 2 sockets
Jun 19 13:18:56 dist-test-slave-mkt3 krb5kdc[22285](info): commencing operation
krb5kdc: starting...
W20260619 13:18:57.202068 388 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:57 dist-test-slave-mkt3 krb5kdc[22285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:18:57.418761 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42601
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:57.727229 22298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:57.727516 22298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:57.727655 22298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:57.734694 22298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:57.734809 22298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:57.734887 22298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:57.734967 22298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:57.735033 22298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:57.744549 22298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:57.745746 22298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:57.747027 22298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:57.757941 22303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:57.757963 22306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:57.757927 22304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:57.758621 22298 server_base.cc:1034] running on GCE node
I20260619 13:18:57.759342 22298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:57.760620 22298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:57.761814 22298 hybrid_clock.cc:648] HybridClock initialized: now 1781875137761771 us; error 33 us; skew 500 ppm
I20260619 13:18:57.762154 22298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:57 dist-test-slave-mkt3 krb5kdc[22285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:57.765002 22298 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:18:57.766350 22298 webserver.cc:460] Webserver started at http://127.0.97.62:40805/ using document root <none> and password file <none>
I20260619 13:18:57.766974 22298 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:57.767089 22298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:57.767361 22298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:57.770164 22298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "531ac0ca87254f65889e2e8ad0bd8c42"
format_stamp: "Formatted at 2026-06-19 13:18:57 on dist-test-slave-mkt3"
server_key: "50957114b5b6a30482b2537f9cfbde81"
server_key_iv: "6e388d40f2a02edeac0509ba62c93fbd"
server_key_version: "encryptionkey@0"
I20260619 13:18:57.770874 22298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "531ac0ca87254f65889e2e8ad0bd8c42"
format_stamp: "Formatted at 2026-06-19 13:18:57 on dist-test-slave-mkt3"
server_key: "50957114b5b6a30482b2537f9cfbde81"
server_key_iv: "6e388d40f2a02edeac0509ba62c93fbd"
server_key_version: "encryptionkey@0"
I20260619 13:18:57.775694 22298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:18:57.778915 22313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:57.780115 22298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:18:57.780265 22298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "531ac0ca87254f65889e2e8ad0bd8c42"
format_stamp: "Formatted at 2026-06-19 13:18:57 on dist-test-slave-mkt3"
server_key: "50957114b5b6a30482b2537f9cfbde81"
server_key_iv: "6e388d40f2a02edeac0509ba62c93fbd"
server_key_version: "encryptionkey@0"
I20260619 13:18:57.780439 22298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:57.785552 22298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:57.789521 22298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:57.789765 22298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:57.806284 22298 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42601
I20260619 13:18:57.806331 22364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42601 every 8 connection(s)
I20260619 13:18:57.807559 22298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:57.809504 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 22298
I20260619 13:18:57.809636 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:57.810120 388 external_mini_cluster.cc:1428] Setting key 7abf5b3e9f9c892ea8987955b6d1f4ab
I20260619 13:18:57.811236 22365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:57.817277 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:57.811799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33963 (local address 127.0.97.62:42601)
0619 13:18:57.812530 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:57.812541 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:57.812581 (+ 40us) server_negotiation.cc:407] Connection header received
0619 13:18:57.814115 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:57.814136 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:57.814474 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:57.814825 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:57.815779 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:57.815785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:57.815798 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:57.815815 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:57.815889 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:57.815909 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:57.816111 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:57.816254 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 13:18:57.816416 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":329,"threads_started":1}
I20260619 13:18:57.820714 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:57.823135 22365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42: Bootstrap starting.
I20260619 13:18:57.826473 22365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:57.827042 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:57.824711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52065 (local address 127.0.97.62:42601)
0619 13:18:57.824842 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:57.824848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:57.825258 (+ 410us) server_negotiation.cc:407] Connection header received
0619 13:18:57.825398 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:57.825404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:57.825449 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:57.825525 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:57.826360 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:57.826365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:57.826370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:18:57.826391 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:57.826427 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:18:57.826440 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:57.826535 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:57.826697 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 13:18:57.826762 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 13:18:57.827729 22365 log.cc:824] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:57.830260 22365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42: No bootstrap required, opened a new log
I20260619 13:18:57.837252 22365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531ac0ca87254f65889e2e8ad0bd8c42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42601 } }
I20260619 13:18:57.837574 22365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:57.837646 22365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 531ac0ca87254f65889e2e8ad0bd8c42, State: Initialized, Role: FOLLOWER
I20260619 13:18:57.838189 22365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531ac0ca87254f65889e2e8ad0bd8c42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42601 } }
I20260619 13:18:57.838380 22365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:57.838474 22365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:57.838601 22365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:57.840826 22365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531ac0ca87254f65889e2e8ad0bd8c42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42601 } }
I20260619 13:18:57.841262 22365 leader_election.cc:302] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 531ac0ca87254f65889e2e8ad0bd8c42; no voters:
I20260619 13:18:57.841739 22365 leader_election.cc:288] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:57.841864 22375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:57.842494 22375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [term 1 LEADER]: Becoming Leader. State: Replica: 531ac0ca87254f65889e2e8ad0bd8c42, State: Running, Role: LEADER
I20260619 13:18:57.842965 22375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531ac0ca87254f65889e2e8ad0bd8c42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42601 } }
I20260619 13:18:57.843619 22365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:57.845433 22376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "531ac0ca87254f65889e2e8ad0bd8c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531ac0ca87254f65889e2e8ad0bd8c42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42601 } } }
I20260619 13:18:57.845405 22377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 531ac0ca87254f65889e2e8ad0bd8c42. Latest consensus state: current_term: 1 leader_uuid: "531ac0ca87254f65889e2e8ad0bd8c42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531ac0ca87254f65889e2e8ad0bd8c42" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42601 } } }
I20260619 13:18:57.845855 22376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:57.845861 22377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:57.848656 22381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:57.853801 22381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:57.862656 22381 catalog_manager.cc:1347] Generated new cluster ID: 48ce61fa65264b0e9476c2a6da00a3de
I20260619 13:18:57.862775 22381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:57.882257 22381 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:57.882747 22381 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:57.889717 22381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 531ac0ca87254f65889e2e8ad0bd8c42: Generated new TSK 0
I20260619 13:18:57.890455 22381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:57.896243 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 22298
W20260619 13:18:57.897578 22360 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33963
W20260619 13:18:57.898280 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 22298: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:18:57.898409 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 22298
2026-06-19T13:18:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1701 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:57 dist-test-slave-mkt3 krb5kdc[22396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:57 dist-test-slave-mkt3 krb5kdc[22396](info): set up 2 sockets
Jun 19 13:18:57 dist-test-slave-mkt3 krb5kdc[22396](info): commencing operation
krb5kdc: starting...
W20260619 13:18:58.906004 388 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:18:59 dist-test-slave-mkt3 krb5kdc[22396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:18:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:18:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:18:59.142879 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44813
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:18:59.468923 22409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:18:59.469208 22409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:18:59.469341 22409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:18:59.476440 22409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:18:59.476539 22409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:18:59.476608 22409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:18:59.476686 22409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:18:59.476740 22409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:18:59.486243 22409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:18:59.487414 22409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:18:59.488637 22409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:18:59.499518 22414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:59.499447 22417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:18:59.499486 22415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:18:59.500589 22409 server_base.cc:1034] running on GCE node
I20260619 13:18:59.501363 22409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:18:59.502774 22409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:18:59.503978 22409 hybrid_clock.cc:648] HybridClock initialized: now 1781875139503923 us; error 50 us; skew 500 ppm
I20260619 13:18:59.504319 22409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:18:59 dist-test-slave-mkt3 krb5kdc[22396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:18:59.507329 22409 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:18:59.508728 22409 webserver.cc:460] Webserver started at http://127.0.97.62:46811/ using document root <none> and password file <none>
I20260619 13:18:59.509408 22409 fs_manager.cc:359] Metadata directory not provided
I20260619 13:18:59.509527 22409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:18:59.509804 22409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:18:59.512738 22409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "f271330812c7493397a00c06862f7f06"
format_stamp: "Formatted at 2026-06-19 13:18:59 on dist-test-slave-mkt3"
server_key: "f6b7f7701b385d64cfaf8c295353688b"
server_key_iv: "530b44e06ac34398457949c272e62a37"
server_key_version: "encryptionkey@0"
I20260619 13:18:59.513463 22409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "f271330812c7493397a00c06862f7f06"
format_stamp: "Formatted at 2026-06-19 13:18:59 on dist-test-slave-mkt3"
server_key: "f6b7f7701b385d64cfaf8c295353688b"
server_key_iv: "530b44e06ac34398457949c272e62a37"
server_key_version: "encryptionkey@0"
I20260619 13:18:59.518385 22409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 13:18:59.521543 22424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:18:59.522779 22409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:18:59.522931 22409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "f271330812c7493397a00c06862f7f06"
format_stamp: "Formatted at 2026-06-19 13:18:59 on dist-test-slave-mkt3"
server_key: "f6b7f7701b385d64cfaf8c295353688b"
server_key_iv: "530b44e06ac34398457949c272e62a37"
server_key_version: "encryptionkey@0"
I20260619 13:18:59.523094 22409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:18:59.534880 22409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:18:59.538666 22409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:18:59.538903 22409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:18:59.555220 22409 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44813
I20260619 13:18:59.555255 22475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44813 every 8 connection(s)
I20260619 13:18:59.556483 22409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:18:59.559994 22476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:18:59.564921 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 22409
I20260619 13:18:59.565097 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:18:59.565593 388 external_mini_cluster.cc:1428] Setting key dc9ddd5a3112774ee585a603797942a1
I20260619 13:18:59.570891 22476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06: Bootstrap starting.
I20260619 13:18:59.574394 22476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06: Neither blocks nor log segments found. Creating new log.
I20260619 13:18:59.575572 22476 log.cc:824] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06: Log is configured to *not* fsync() on all Append() calls
I20260619 13:18:59.578262 22476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06: No bootstrap required, opened a new log
Jun 19 13:18:59 dist-test-slave-mkt3 krb5kdc[22396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:18:59.588124 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:59.567462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58257 (local address 127.0.97.62:44813)
0619 13:18:59.568224 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:59.568235 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:59.568278 (+ 43us) server_negotiation.cc:407] Connection header received
0619 13:18:59.570011 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:59.570034 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:59.570408 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:59.570729 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:59.572369 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:59.573636 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:59.575370 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:59.576156 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:59.576339 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:59.579977 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:59.579991 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:59.580009 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:59.580053 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:59.583918 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:59.584503 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:59.584512 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:59.584520 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:59.584730 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:59.585040 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:59.585045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:59.585048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:59.585730 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:59.586047 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:59.586349 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 13:18:59.586793 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":331,"threads_started":1}
I20260619 13:18:59.588192 22476 raft_consensus.cc:348] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f271330812c7493397a00c06862f7f06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44813 } }
I20260619 13:18:59.588625 22476 raft_consensus.cc:374] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:18:59.588752 22476 raft_consensus.cc:729] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f271330812c7493397a00c06862f7f06, State: Initialized, Role: FOLLOWER
I20260619 13:18:59.589388 22476 consensus_queue.cc:260] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f271330812c7493397a00c06862f7f06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44813 } }
I20260619 13:18:59.589649 22476 raft_consensus.cc:388] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:18:59.589790 22476 raft_consensus.cc:482] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:18:59.589980 22476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:18:59.591953 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:18:59.592521 22476 raft_consensus.cc:504] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f271330812c7493397a00c06862f7f06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44813 } }
I20260619 13:18:59.592970 22476 leader_election.cc:302] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f271330812c7493397a00c06862f7f06; no voters:
I20260619 13:18:59.593508 22476 leader_election.cc:288] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:18:59.593667 22481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:18:59.595228 22476 sys_catalog.cc:564] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:18:59.598470 22481 raft_consensus.cc:686] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [term 1 LEADER]: Becoming Leader. State: Replica: f271330812c7493397a00c06862f7f06, State: Running, Role: LEADER
I20260619 13:18:59.599133 22481 consensus_queue.cc:237] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f271330812c7493397a00c06862f7f06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44813 } }
I20260619 13:18:59.601828 22482 sys_catalog.cc:455] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f271330812c7493397a00c06862f7f06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f271330812c7493397a00c06862f7f06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44813 } } }
I20260619 13:18:59.602239 22482 sys_catalog.cc:458] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:59.602746 22483 sys_catalog.cc:455] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f271330812c7493397a00c06862f7f06. Latest consensus state: current_term: 1 leader_uuid: "f271330812c7493397a00c06862f7f06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f271330812c7493397a00c06862f7f06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44813 } } }
I20260619 13:18:59.602993 22483 sys_catalog.cc:458] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06 [sys.catalog]: This master's current role is: LEADER
I20260619 13:18:59.603051 22493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:18:59.613962 22493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:18:59.636315 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:18:59.607220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52723 (local address 127.0.97.62:44813)
0619 13:18:59.607419 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 13:18:59.607425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:18:59.618926 (+ 11501us) server_negotiation.cc:407] Connection header received
0619 13:18:59.619159 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:18:59.619165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:18:59.619245 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:18:59.619347 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:18:59.621562 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:59.622777 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:59.624761 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:18:59.625475 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:18:59.625571 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:18:59.629001 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:18:59.629009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:18:59.629015 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:18:59.629056 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:18:59.632860 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:59.633715 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:59.633722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:59.633726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:59.633905 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:18:59.634360 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:18:59.634366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:18:59.634370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:18:59.634798 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:18:59.635043 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 13:18:59.635666 (+ 623us) server_negotiation.cc:299] Negotiation successful
0619 13:18:59.635902 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 13:18:59.639430 22493 catalog_manager.cc:1347] Generated new cluster ID: 3dd7636d14964c0ab860d7bdae8bf4b7
I20260619 13:18:59.639540 22493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:18:59.648752 22493 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:18:59.650331 22493 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:18:59.656363 22493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f271330812c7493397a00c06862f7f06: Generated new TSK 0
I20260619 13:18:59.657063 22493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:18:59.662324 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 22409
I20260619 13:18:59.663683 22471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:18:59.751448 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 22409
2026-06-19T13:18:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1853 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:18:59 dist-test-slave-mkt3 krb5kdc[22509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:18:59 dist-test-slave-mkt3 krb5kdc[22509](info): set up 2 sockets
Jun 19 13:18:59 dist-test-slave-mkt3 krb5kdc[22509](info): commencing operation
krb5kdc: starting...
W20260619 13:19:00.753894 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:19:00 dist-test-slave-mkt3 krb5kdc[22509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:19:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:19:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:19:00.976140 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43663
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:19:01.288792 22522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:19:01.289077 22522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:19:01.289209 22522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:19:01.296252 22522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:19:01.296350 22522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:19:01.296406 22522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:19:01.296487 22522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:19:01.296541 22522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:19:01.306066 22522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:19:01.307268 22522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:19:01.308534 22522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:19:01.319221 22528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:01.319254 22530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:01.319190 22527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:19:01.319870 22522 server_base.cc:1034] running on GCE node
I20260619 13:19:01.320598 22522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:19:01.321910 22522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:19:01.323139 22522 hybrid_clock.cc:648] HybridClock initialized: now 1781875141323090 us; error 40 us; skew 500 ppm
I20260619 13:19:01.323475 22522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:19:01 dist-test-slave-mkt3 krb5kdc[22509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:19:01.330482 22522 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:19:01.331925 22522 webserver.cc:460] Webserver started at http://127.0.97.62:46711/ using document root <none> and password file <none>
I20260619 13:19:01.332568 22522 fs_manager.cc:359] Metadata directory not provided
I20260619 13:19:01.332666 22522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:19:01.332906 22522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:19:01.335961 22522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "9fea8aa0ce17476eafb704d658771696"
format_stamp: "Formatted at 2026-06-19 13:19:01 on dist-test-slave-mkt3"
server_key: "4ac3da9ee01a119a7c5fb8d8d7f6e98e"
server_key_iv: "6e9045f0bc3c51b1ffd211677edb2ae2"
server_key_version: "encryptionkey@0"
I20260619 13:19:01.336678 22522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "9fea8aa0ce17476eafb704d658771696"
format_stamp: "Formatted at 2026-06-19 13:19:01 on dist-test-slave-mkt3"
server_key: "4ac3da9ee01a119a7c5fb8d8d7f6e98e"
server_key_iv: "6e9045f0bc3c51b1ffd211677edb2ae2"
server_key_version: "encryptionkey@0"
I20260619 13:19:01.341563 22522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 13:19:01.344851 22537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:19:01.346180 22522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 13:19:01.346314 22522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "9fea8aa0ce17476eafb704d658771696"
format_stamp: "Formatted at 2026-06-19 13:19:01 on dist-test-slave-mkt3"
server_key: "4ac3da9ee01a119a7c5fb8d8d7f6e98e"
server_key_iv: "6e9045f0bc3c51b1ffd211677edb2ae2"
server_key_version: "encryptionkey@0"
I20260619 13:19:01.346472 22522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:19:01.362912 22522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:19:01.366494 22522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:19:01.366711 22522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:19:01.383040 22522 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43663
I20260619 13:19:01.383090 22588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43663 every 8 connection(s)
I20260619 13:19:01.384325 22522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:19:01.386816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 22522
I20260619 13:19:01.386976 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:19:01.387382 388 external_mini_cluster.cc:1428] Setting key 60e9f0b4ca303bb0567592f2fddcc3a4
I20260619 13:19:01.387890 22589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:19:01.401906 22589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696: Bootstrap starting.
I20260619 13:19:01.401909 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:01.389059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34987 (local address 127.0.97.62:43663)
0619 13:19:01.389809 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:01.389823 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:01.389894 (+ 71us) server_negotiation.cc:407] Connection header received
0619 13:19:01.392296 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:01.392325 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:01.392840 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:01.393268 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:01.393772 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:01.395724 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:01.397037 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:01.398047 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:01.398231 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:01.399246 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:01.399255 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:01.399276 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:19:01.399970 (+ 694us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:01.400073 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:19:01.400102 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:01.400198 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:01.400404 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 13:19:01.400647 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":305,"threads_started":1}
I20260619 13:19:01.405339 22589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696: Neither blocks nor log segments found. Creating new log.
I20260619 13:19:01.405539 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:19:01.406373 22589 log.cc:824] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696: Log is configured to *not* fsync() on all Append() calls
I20260619 13:19:01.408982 22589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696: No bootstrap required, opened a new log
I20260619 13:19:01.415594 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:01.409722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33071 (local address 127.0.97.62:43663)
0619 13:19:01.409897 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:01.409902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:01.410396 (+ 494us) server_negotiation.cc:407] Connection header received
0619 13:19:01.410561 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:01.410570 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:01.410638 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:01.410724 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:01.411195 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:01.412111 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:01.413364 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:01.413833 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:01.413978 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:01.414924 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:01.414929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:01.414932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:19:01.414951 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:01.414986 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:19:01.415001 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:01.415087 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:01.415204 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 13:19:01.415296 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 13:19:01.417112 22589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fea8aa0ce17476eafb704d658771696" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43663 } }
I20260619 13:19:01.417461 22589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:19:01.417549 22589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fea8aa0ce17476eafb704d658771696, State: Initialized, Role: FOLLOWER
I20260619 13:19:01.418149 22589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fea8aa0ce17476eafb704d658771696" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43663 } }
I20260619 13:19:01.418367 22589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:19:01.418490 22589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:19:01.418635 22589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:19:01.421037 22589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fea8aa0ce17476eafb704d658771696" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43663 } }
I20260619 13:19:01.421489 22589 leader_election.cc:302] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fea8aa0ce17476eafb704d658771696; no voters:
I20260619 13:19:01.422040 22589 leader_election.cc:288] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:19:01.422122 22599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:19:01.422755 22599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [term 1 LEADER]: Becoming Leader. State: Replica: 9fea8aa0ce17476eafb704d658771696, State: Running, Role: LEADER
I20260619 13:19:01.423274 22599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fea8aa0ce17476eafb704d658771696" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43663 } }
I20260619 13:19:01.423961 22589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:19:01.425947 22600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fea8aa0ce17476eafb704d658771696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fea8aa0ce17476eafb704d658771696" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43663 } } }
I20260619 13:19:01.425765 22601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fea8aa0ce17476eafb704d658771696. Latest consensus state: current_term: 1 leader_uuid: "9fea8aa0ce17476eafb704d658771696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fea8aa0ce17476eafb704d658771696" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43663 } } }
I20260619 13:19:01.426338 22601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [sys.catalog]: This master's current role is: LEADER
I20260619 13:19:01.426339 22600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696 [sys.catalog]: This master's current role is: LEADER
I20260619 13:19:01.427758 22606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:19:01.434530 22606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:19:01.443602 22606 catalog_manager.cc:1347] Generated new cluster ID: 00cd8ae546114b3f9ded00a102d43336
I20260619 13:19:01.443696 22606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:19:01.461787 22606 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:19:01.463076 22606 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:19:01.473819 22606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fea8aa0ce17476eafb704d658771696: Generated new TSK 0
I20260619 13:19:01.474658 22606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:19:01.490984 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 22522
W20260619 13:19:01.492077 22584 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34987
W20260619 13:19:01.492682 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 22522: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:19:01.492795 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 22522
2026-06-19T13:19:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1741 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:19:01 dist-test-slave-mkt3 krb5kdc[22620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:19:01 dist-test-slave-mkt3 krb5kdc[22620](info): set up 2 sockets
Jun 19 13:19:01 dist-test-slave-mkt3 krb5kdc[22620](info): commencing operation
krb5kdc: starting...
W20260619 13:19:02.489975 388 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:19:02 dist-test-slave-mkt3 krb5kdc[22620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:19:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:19:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:19:02.708642 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40809
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:19:03.021361 22633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:19:03.021690 22633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:19:03.021829 22633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:19:03.029057 22633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:19:03.029160 22633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:19:03.029213 22633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:19:03.029271 22633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:19:03.029312 22633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:19:03.038932 22633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:19:03.040064 22633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:19:03.041270 22633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:19:03.052006 22641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:03.052054 22639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:03.052083 22638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:19:03.052785 22633 server_base.cc:1034] running on GCE node
I20260619 13:19:03.053498 22633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:19:03.054818 22633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:19:03.056001 22633 hybrid_clock.cc:648] HybridClock initialized: now 1781875143055948 us; error 51 us; skew 500 ppm
I20260619 13:19:03.056314 22633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:19:03 dist-test-slave-mkt3 krb5kdc[22620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:19:03.059226 22633 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:19:03.060506 22633 webserver.cc:460] Webserver started at http://127.0.97.62:41363/ using document root <none> and password file <none>
I20260619 13:19:03.061159 22633 fs_manager.cc:359] Metadata directory not provided
I20260619 13:19:03.061271 22633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:19:03.061537 22633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:19:03.064405 22633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "5c62e392ddf948619a659ed7f27dd347"
format_stamp: "Formatted at 2026-06-19 13:19:03 on dist-test-slave-mkt3"
server_key: "4f20bb95a3631e21420effbab6b9b289"
server_key_iv: "e67ad59702a4f5f30f864fec053267a2"
server_key_version: "encryptionkey@0"
I20260619 13:19:03.065129 22633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "5c62e392ddf948619a659ed7f27dd347"
format_stamp: "Formatted at 2026-06-19 13:19:03 on dist-test-slave-mkt3"
server_key: "4f20bb95a3631e21420effbab6b9b289"
server_key_iv: "e67ad59702a4f5f30f864fec053267a2"
server_key_version: "encryptionkey@0"
I20260619 13:19:03.070142 22633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 13:19:03.073370 22648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:19:03.074585 22633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 13:19:03.074743 22633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "5c62e392ddf948619a659ed7f27dd347"
format_stamp: "Formatted at 2026-06-19 13:19:03 on dist-test-slave-mkt3"
server_key: "4f20bb95a3631e21420effbab6b9b289"
server_key_iv: "e67ad59702a4f5f30f864fec053267a2"
server_key_version: "encryptionkey@0"
I20260619 13:19:03.074913 22633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:19:03.080224 22633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:19:03.084116 22633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:19:03.084367 22633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:19:03.100034 22633 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40809
I20260619 13:19:03.100075 22699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40809 every 8 connection(s)
I20260619 13:19:03.101325 22633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:19:03.104821 22700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:19:03.108615 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 22633
I20260619 13:19:03.108775 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:19:03.109191 388 external_mini_cluster.cc:1428] Setting key 650a91bf8949340b6824d5909c9398a3
I20260619 13:19:03.116365 22700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347: Bootstrap starting.
I20260619 13:19:03.118355 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:03.110914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58259 (local address 127.0.97.62:40809)
0619 13:19:03.111678 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:03.111693 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:03.111743 (+ 50us) server_negotiation.cc:407] Connection header received
0619 13:19:03.114593 (+ 2850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:03.114621 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:03.115062 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:03.115484 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:03.116438 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:03.116447 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:03.116463 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:19:03.116480 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:03.116550 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:19:03.116582 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:03.116814 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:03.116944 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 13:19:03.117123 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":310,"threads_started":1}
I20260619 13:19:03.119998 22700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347: Neither blocks nor log segments found. Creating new log.
I20260619 13:19:03.121202 22700 log.cc:824] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347: Log is configured to *not* fsync() on all Append() calls
I20260619 13:19:03.122076 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:19:03.123886 22700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347: No bootstrap required, opened a new log
I20260619 13:19:03.128140 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:03.126035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40569 (local address 127.0.97.62:40809)
0619 13:19:03.126206 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:03.126211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:03.126532 (+ 321us) server_negotiation.cc:407] Connection header received
0619 13:19:03.126644 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:03.126648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:03.126710 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:03.126812 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:03.127571 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:03.127577 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:03.127581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 13:19:03.127599 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:03.127642 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 13:19:03.127669 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:03.127768 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:03.127827 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 13:19:03.127877 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:19:03.132406 22700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c62e392ddf948619a659ed7f27dd347" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40809 } }
I20260619 13:19:03.132732 22700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 13:19:03.132803 22700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c62e392ddf948619a659ed7f27dd347, State: Initialized, Role: FOLLOWER
I20260619 13:19:03.133273 22700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c62e392ddf948619a659ed7f27dd347" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40809 } }
I20260619 13:19:03.133463 22700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 13:19:03.133558 22700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 13:19:03.133682 22700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:19:03.136075 22700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c62e392ddf948619a659ed7f27dd347" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40809 } }
I20260619 13:19:03.136518 22700 leader_election.cc:302] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c62e392ddf948619a659ed7f27dd347; no voters:
I20260619 13:19:03.136988 22700 leader_election.cc:288] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 13:19:03.137109 22710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:19:03.137735 22710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [term 1 LEADER]: Becoming Leader. State: Replica: 5c62e392ddf948619a659ed7f27dd347, State: Running, Role: LEADER
I20260619 13:19:03.138303 22710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c62e392ddf948619a659ed7f27dd347" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40809 } }
I20260619 13:19:03.138921 22700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:19:03.140960 22712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c62e392ddf948619a659ed7f27dd347. Latest consensus state: current_term: 1 leader_uuid: "5c62e392ddf948619a659ed7f27dd347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c62e392ddf948619a659ed7f27dd347" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40809 } } }
I20260619 13:19:03.141017 22711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c62e392ddf948619a659ed7f27dd347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c62e392ddf948619a659ed7f27dd347" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40809 } } }
I20260619 13:19:03.141348 22712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [sys.catalog]: This master's current role is: LEADER
I20260619 13:19:03.141347 22711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347 [sys.catalog]: This master's current role is: LEADER
I20260619 13:19:03.147073 22717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:19:03.152022 22717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:19:03.160463 22717 catalog_manager.cc:1347] Generated new cluster ID: 79073e3c69534924bad1541746527865
I20260619 13:19:03.160578 22717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:19:03.177997 22717 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:19:03.178409 22717 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:19:03.186558 22717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c62e392ddf948619a659ed7f27dd347: Generated new TSK 0
I20260619 13:19:03.187306 22717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:19:03.193624 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 22633
W20260619 13:19:03.194705 22695 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58259
W20260619 13:19:03.195397 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 22633: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 13:19:03.195523 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 22633
2026-06-19T13:19:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1702 ms)
[----------] 18 tests from AuthTokenIssuingTest (31967 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:19:03 dist-test-slave-mkt3 krb5kdc[22731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:19:03 dist-test-slave-mkt3 krb5kdc[22731](info): set up 2 sockets
Jun 19 13:19:03 dist-test-slave-mkt3 krb5kdc[22731](info): commencing operation
krb5kdc: starting...
W20260619 13:19:04.206168 388 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:19:04 dist-test-slave-mkt3 krb5kdc[22731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:19:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:19:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:19:04.431283 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44877
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44877,127.0.97.61:43615,127.0.97.60:41767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:19:04.743306 22744 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:19:04.743651 22744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:19:04.743754 22744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:19:04.743867 22744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:19:04.751000 22744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:19:04.751106 22744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:19:04.751195 22744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:19:04.751257 22744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:19:04.751340 22744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:19:04.760883 22744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44877,127.0.97.61:43615,127.0.97.60:41767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:19:04.762216 22744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:19:04.763468 22744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:19:04.774145 22750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:04.774235 22749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:04.774215 22752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:19:04.775040 22744 server_base.cc:1034] running on GCE node
I20260619 13:19:04.775731 22744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:19:04.776960 22744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:19:04.778177 22744 hybrid_clock.cc:648] HybridClock initialized: now 1781875144778117 us; error 39 us; skew 500 ppm
I20260619 13:19:04.778484 22744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:19:04 dist-test-slave-mkt3 krb5kdc[22731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:19:04.781380 22744 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:19:04.782747 22744 webserver.cc:460] Webserver started at http://127.0.97.62:33817/ using document root <none> and password file <none>
I20260619 13:19:04.783393 22744 fs_manager.cc:359] Metadata directory not provided
I20260619 13:19:04.783488 22744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:19:04.783766 22744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:19:04.786586 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "b94ff5be567f47e2b516abcf5fbb7bb1"
format_stamp: "Formatted at 2026-06-19 13:19:04 on dist-test-slave-mkt3"
server_key: "b572848d02ce33e7b2590a42144a9774"
server_key_iv: "7146f5766d9c02b7ee573e44e0d82871"
server_key_version: "encryptionkey@0"
I20260619 13:19:04.787317 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "b94ff5be567f47e2b516abcf5fbb7bb1"
format_stamp: "Formatted at 2026-06-19 13:19:04 on dist-test-slave-mkt3"
server_key: "b572848d02ce33e7b2590a42144a9774"
server_key_iv: "7146f5766d9c02b7ee573e44e0d82871"
server_key_version: "encryptionkey@0"
I20260619 13:19:04.792127 22744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:19:04.795296 22759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:19:04.796536 22744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 13:19:04.796692 22744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "b94ff5be567f47e2b516abcf5fbb7bb1"
format_stamp: "Formatted at 2026-06-19 13:19:04 on dist-test-slave-mkt3"
server_key: "b572848d02ce33e7b2590a42144a9774"
server_key_iv: "7146f5766d9c02b7ee573e44e0d82871"
server_key_version: "encryptionkey@0"
I20260619 13:19:04.796880 22744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:19:04.813284 22744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:19:04.817297 22744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:19:04.817538 22744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:19:04.834235 22744 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44877
I20260619 13:19:04.834254 22810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44877 every 8 connection(s)
I20260619 13:19:04.835479 22744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:19:04.838929 22811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:19:04.842399 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 22744
I20260619 13:19:04.842618 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:19:04.843379 388 external_mini_cluster.cc:1428] Setting key 9f58aea728e419cd987320683e60bd5e
I20260619 13:19:04.846633 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:19:04 dist-test-slave-mkt3 krb5kdc[22731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260619 13:19:04.868530 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:04.850043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.0.97.62:44877)
0619 13:19:04.850808 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:04.850821 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:04.851380 (+ 559us) server_negotiation.cc:407] Connection header received
0619 13:19:04.853232 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:04.853237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:04.853453 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:04.853625 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:04.854887 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:04.856047 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:04.857366 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:04.857936 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:04.858109 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:04.861617 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:04.861625 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:04.861636 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:04.861670 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:04.864259 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:04.865058 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:04.865064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:04.865066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:04.865246 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:04.865590 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:04.865595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:04.865598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:04.866142 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:04.866323 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:04.866835 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 13:19:04.867129 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":544,"thread_start_us":327,"threads_started":1}
I20260619 13:19:04.868541 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:04.850043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44877 (local address 127.0.0.1:41624)
0619 13:19:04.851202 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:04.851243 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:04.852897 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:04.853671 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:04.853677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:04.854355 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:04.854693 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:04.854710 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:04.856197 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:04.856200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:04.857240 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:04.857252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:04.858150 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:04.858158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:04.858366 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:04.858945 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:04.858965 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:04.861405 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:04.864394 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:04.864408 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:04.864420 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:04.864946 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:04.865342 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:04.865347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:04.865351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:04.865496 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:04.866273 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:04.866281 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:04.866479 (+ 198us) client_negotiation.cc:769] Sending connection context
0619 13:19:04.866705 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 13:19:04.867248 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"mutex_wait_us":145,"spinlock_wait_cycles":15616,"thread_start_us":687,"threads_started":1}
I20260619 13:19:04.871591 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:19:04.872558 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:04.872188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43615 (local address 127.0.0.1:34628)
0619 13:19:04.872345 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:04.872443 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 13:19:04.872815 22762 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
I20260619 13:19:04.874221 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:04.873915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43615 (local address 127.0.0.1:34634)
0619 13:19:04.874035 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:04.874102 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 13:19:04.874396 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43615: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 13:19:04.893913 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43615
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44877,127.0.97.61:43615,127.0.97.60:41767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:19:04.923702 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } attempt: 1
I20260619 13:19:04.924835 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:04.924405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43615 (local address 127.0.0.1:34648)
0619 13:19:04.924603 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:04.924709 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 13:19:04.925773 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:04.925432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43615 (local address 127.0.0.1:34650)
0619 13:19:04.925571 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:04.925644 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 13:19:04.925978 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43615: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
I20260619 13:19:04.994194 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } attempt: 2
I20260619 13:19:04.995324 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:04.994816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43615 (local address 127.0.0.1:34652)
0619 13:19:04.995092 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:04.995188 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 13:19:04.996237 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:04.995902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43615 (local address 127.0.0.1:34658)
0619 13:19:04.996050 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:04.996113 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 13:19:04.996403 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43615: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
I20260619 13:19:05.087631 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } attempt: 3
I20260619 13:19:05.088696 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.088255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43615 (local address 127.0.0.1:34674)
0619 13:19:05.088440 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.088535 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 13:19:05.089816 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.089370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43615 (local address 127.0.0.1:34690)
0619 13:19:05.089579 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.089677 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 13:19:05.090087 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43615: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
W20260619 13:19:05.224722 22818 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:19:05.225040 22818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:19:05.225113 22818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:19:05.225194 22818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:19:05.232503 22818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:19:05.232616 22818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:19:05.232673 22818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:19:05.232726 22818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:19:05.232792 22818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:19:05.233322 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } attempt: 4
I20260619 13:19:05.234478 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.233971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43615 (local address 127.0.0.1:34702)
0619 13:19:05.234259 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.234347 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260619 13:19:05.235538 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.235172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43615 (local address 127.0.0.1:34714)
0619 13:19:05.235315 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.235401 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 13:19:05.235736 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43615: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43615: connect: Connection refused (error 111)
I20260619 13:19:05.243116 22818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44877,127.0.97.61:43615,127.0.97.60:41767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:19:05.244359 22818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:19:05.245630 22818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:19:05.257345 22823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:05.257593 22826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:05.257788 22824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:19:05.258740 22818 server_base.cc:1034] running on GCE node
I20260619 13:19:05.259708 22818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:19:05.261323 22818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:19:05.262547 22818 hybrid_clock.cc:648] HybridClock initialized: now 1781875145262482 us; error 52 us; skew 500 ppm
I20260619 13:19:05.262890 22818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:19:05 dist-test-slave-mkt3 krb5kdc[22731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:19:05.266695 22818 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 13:19:05.268478 22818 webserver.cc:460] Webserver started at http://127.0.97.61:37307/ using document root <none> and password file <none>
I20260619 13:19:05.269387 22818 fs_manager.cc:359] Metadata directory not provided
I20260619 13:19:05.269507 22818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:19:05.269917 22818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:19:05.273039 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/data/instance:
uuid: "4783b70262834e15a4ce574bd9bdce00"
format_stamp: "Formatted at 2026-06-19 13:19:05 on dist-test-slave-mkt3"
server_key: "66be3a381d484e085d604218b248d94c"
server_key_iv: "956e1a08a9e8ee1ba246a9a0fa67fe9c"
server_key_version: "encryptionkey@0"
I20260619 13:19:05.273793 22818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/wal/instance:
uuid: "4783b70262834e15a4ce574bd9bdce00"
format_stamp: "Formatted at 2026-06-19 13:19:05 on dist-test-slave-mkt3"
server_key: "66be3a381d484e085d604218b248d94c"
server_key_iv: "956e1a08a9e8ee1ba246a9a0fa67fe9c"
server_key_version: "encryptionkey@0"
I20260619 13:19:05.278746 22818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 13:19:05.281962 22833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:19:05.283243 22818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 13:19:05.283393 22818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/wal
uuid: "4783b70262834e15a4ce574bd9bdce00"
format_stamp: "Formatted at 2026-06-19 13:19:05 on dist-test-slave-mkt3"
server_key: "66be3a381d484e085d604218b248d94c"
server_key_iv: "956e1a08a9e8ee1ba246a9a0fa67fe9c"
server_key_version: "encryptionkey@0"
I20260619 13:19:05.283587 22818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:19:05.299999 22818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:19:05.303987 22818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:19:05.304217 22818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:19:05.321074 22818 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43615
I20260619 13:19:05.321094 22884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43615 every 8 connection(s)
I20260619 13:19:05.322363 22818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/data/info.pb
I20260619 13:19:05.325934 22885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:19:05.327332 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 22818
I20260619 13:19:05.327494 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-1/wal/instance
I20260619 13:19:05.327924 388 external_mini_cluster.cc:1428] Setting key 4c94101237626422774a68329862f366
I20260619 13:19:05.332248 22885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:19:05 dist-test-slave-mkt3 krb5kdc[22731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260619 13:19:05.352924 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.334682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.0.97.62:44877)
0619 13:19:05.334841 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:05.334846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:05.336235 (+ 1389us) server_negotiation.cc:407] Connection header received
0619 13:19:05.338677 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:05.338682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:05.338752 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:05.338842 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:05.340597 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.341272 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.343271 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.343680 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.343776 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.347367 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:05.347372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:05.347375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:05.347407 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:05.349920 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.350609 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.350613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.350615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.350732 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.351104 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.351107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.351109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.351381 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:05.351510 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:05.352282 (+ 772us) server_negotiation.cc:299] Negotiation successful
0619 13:19:05.352568 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 13:19:05.353742 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.335300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44877 (local address 127.0.0.1:41632)
0619 13:19:05.336066 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.336112 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:05.338364 (+ 2252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:05.339068 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:05.339079 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:05.339749 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:05.340436 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:05.340458 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.341409 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.341413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:05.343133 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:05.343147 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.343884 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.343892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:05.344152 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.344808 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:05.344835 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:05.347144 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:05.350048 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:05.350057 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:05.350073 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:05.350482 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:05.350830 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:05.350835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:05.350839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:05.351000 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:05.351501 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:05.351508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:05.351919 (+ 411us) client_negotiation.cc:769] Sending connection context
0619 13:19:05.352139 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 13:19:05.352606 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":318,"threads_started":1}
I20260619 13:19:05.355415 22885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:19:05 dist-test-slave-mkt3 krb5kdc[22731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:19:05.369247 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.355948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43615 (local address 127.0.0.1:34716)
0619 13:19:05.356123 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.356140 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:05.356238 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:05.357187 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:05.357191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:05.357505 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:05.357628 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:05.357635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.359433 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.359438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:05.360313 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:05.360328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.361100 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.361106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:05.361293 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.361787 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:05.361807 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:05.364028 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:05.366860 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:05.366865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:05.366870 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:05.367361 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:05.367688 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:05.367692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:05.367695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:05.367828 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:05.368350 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:05.368355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:05.368496 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 13:19:05.368626 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 13:19:05.368877 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 13:19:05.369261 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.356189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.0.97.61:43615)
0619 13:19:05.356735 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:05.356743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:05.356770 (+ 27us) server_negotiation.cc:407] Connection header received
0619 13:19:05.356821 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:05.356828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:05.357028 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:05.357166 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:05.357787 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.359237 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.360442 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.360954 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.361056 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.364227 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:05.364237 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:05.364249 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:05.364276 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:05.366745 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.367474 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.367480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.367482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.367598 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.367916 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.367919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.367921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.368226 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:05.368372 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:05.368751 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 13:19:05.368978 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":391,"thread_start_us":255,"threads_started":1}
I20260619 13:19:05.371739 22885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:19:05.372817 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.372444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46378)
0619 13:19:05.372586 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.372688 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 13:19:05.373085 22837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
I20260619 13:19:05.374498 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.374189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46384)
0619 13:19:05.374338 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.374407 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 13:19:05.374684 22885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41767: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 13:19:05.380779 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41767
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44877,127.0.97.61:43615,127.0.97.60:41767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:19:05.423954 22885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } attempt: 1
I20260619 13:19:05.424975 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.424560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46400)
0619 13:19:05.424739 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.424825 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 13:19:05.426040 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.425680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46410)
0619 13:19:05.425795 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.425910 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 13:19:05.426287 22885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41767: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
I20260619 13:19:05.494498 22885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } attempt: 2
I20260619 13:19:05.495620 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.495162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46412)
0619 13:19:05.495348 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.495455 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 13:19:05.496522 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.496238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46418)
0619 13:19:05.496346 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.496413 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 13:19:05.496733 22885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41767: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
I20260619 13:19:05.534984 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } attempt: 5
Jun 19 13:19:05 dist-test-slave-mkt3 krb5kdc[22731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:19:05.547423 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.535748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43615 (local address 127.0.0.1:34726)
0619 13:19:05.535961 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.535981 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:05.536147 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:05.536511 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:05.536517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:05.536907 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:05.537075 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:05.537095 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.538156 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.538161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:05.539029 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:05.539041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.539751 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.539757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:05.539922 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.540523 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:05.540540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:05.542505 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:05.545192 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:05.545198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:05.545202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:05.545555 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:05.545904 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:05.545909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:05.545914 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:05.546043 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:05.546535 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:05.546540 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:05.546676 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 13:19:05.546791 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 13:19:05.547053 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 13:19:05.547423 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.535793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.0.97.61:43615)
0619 13:19:05.535976 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:05.535982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:05.536063 (+ 81us) server_negotiation.cc:407] Connection header received
0619 13:19:05.536276 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:05.536281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:05.536360 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:05.536445 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:05.537246 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.537999 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.539185 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.539615 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.539725 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.542682 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:05.542687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:05.542690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:05.542721 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:05.545071 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.545662 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.545667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.545669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.545785 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.546146 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.546151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.546155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.546427 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:05.546553 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:05.546906 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 13:19:05.547112 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 13:19:05.548717 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:19:05.549752 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.549369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46426)
0619 13:19:05.549535 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.549618 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 13:19:05.550601 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.550302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46440)
0619 13:19:05.550428 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.550499 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 13:19:05.550777 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41767: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
I20260619 13:19:05.587939 22885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } attempt: 3
I20260619 13:19:05.589092 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.588527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46454)
0619 13:19:05.588756 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.588878 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 13:19:05.590185 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.589687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46460)
0619 13:19:05.589949 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.590046 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260619 13:19:05.590420 22885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41767: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
I20260619 13:19:05.601954 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } attempt: 1
I20260619 13:19:05.602854 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.602455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46466)
0619 13:19:05.602626 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.602706 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 13:19:05.603780 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.603493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46474)
0619 13:19:05.603615 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.603680 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 13:19:05.603969 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41767: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
I20260619 13:19:05.672176 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } attempt: 2
I20260619 13:19:05.673367 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.672884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46476)
0619 13:19:05.673111 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.673202 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 13:19:05.674306 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.674022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46482)
0619 13:19:05.674136 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.674203 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 13:19:05.674506 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41767: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
W20260619 13:19:05.696213 22892 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:19:05.696489 22892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:19:05.696557 22892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:19:05.696633 22892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:19:05.703645 22892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:19:05.703720 22892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:19:05.703774 22892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:19:05.703822 22892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:19:05.703872 22892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:19:05.713042 22892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44877,127.0.97.61:43615,127.0.97.60:41767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:19:05.714172 22892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:19:05.715334 22892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:19:05.726279 22900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:05.726341 22897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:05.726320 22898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:19:05.727022 22892 server_base.cc:1034] running on GCE node
I20260619 13:19:05.727705 22892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:19:05.728931 22892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:19:05.730144 22892 hybrid_clock.cc:648] HybridClock initialized: now 1781875145730060 us; error 45 us; skew 500 ppm
I20260619 13:19:05.730453 22892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:19:05 dist-test-slave-mkt3 krb5kdc[22731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:19:05.733407 22892 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 13:19:05.733639 22885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } attempt: 4
I20260619 13:19:05.734704 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.734255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46496)
0619 13:19:05.734463 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.734543 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 13:19:05.735100 22892 webserver.cc:460] Webserver started at http://127.0.97.60:41971/ using document root <none> and password file <none>
I20260619 13:19:05.735666 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.735358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46504)
0619 13:19:05.735486 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.735562 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 13:19:05.735862 22885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41767: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
I20260619 13:19:05.735908 22892 fs_manager.cc:359] Metadata directory not provided
I20260619 13:19:05.736001 22892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:19:05.736300 22892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:19:05.739184 22892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/data/instance:
uuid: "c16583fa101a47a6a483ed5cc92e828f"
format_stamp: "Formatted at 2026-06-19 13:19:05 on dist-test-slave-mkt3"
server_key: "b4eb6c250aeaf5631017af1d5fc9e4a5"
server_key_iv: "95b0416ba043f25d137f9c9e5fff89ee"
server_key_version: "encryptionkey@0"
I20260619 13:19:05.739910 22892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/wal/instance:
uuid: "c16583fa101a47a6a483ed5cc92e828f"
format_stamp: "Formatted at 2026-06-19 13:19:05 on dist-test-slave-mkt3"
server_key: "b4eb6c250aeaf5631017af1d5fc9e4a5"
server_key_iv: "95b0416ba043f25d137f9c9e5fff89ee"
server_key_version: "encryptionkey@0"
I20260619 13:19:05.744716 22892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 13:19:05.747843 22907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:19:05.749079 22892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 13:19:05.749229 22892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/wal
uuid: "c16583fa101a47a6a483ed5cc92e828f"
format_stamp: "Formatted at 2026-06-19 13:19:05 on dist-test-slave-mkt3"
server_key: "b4eb6c250aeaf5631017af1d5fc9e4a5"
server_key_iv: "95b0416ba043f25d137f9c9e5fff89ee"
server_key_version: "encryptionkey@0"
I20260619 13:19:05.749410 22892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:19:05.763029 22892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:19:05.767007 22892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:19:05.767256 22892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:19:05.780711 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } attempt: 3
I20260619 13:19:05.782001 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.781358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46514)
0619 13:19:05.781709 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.781815 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260619 13:19:05.783047 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.782675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46518)
0619 13:19:05.782874 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.782945 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260619 13:19:05.783267 22811 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41767: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41767: connect: Connection refused (error 111)
I20260619 13:19:05.784132 22892 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41767
I20260619 13:19:05.784149 22958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41767 every 8 connection(s)
I20260619 13:19:05.785362 22892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/data/info.pb
I20260619 13:19:05.788961 22959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:19:05.791863 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 22892
I20260619 13:19:05.792019 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781874807295266-388-0/minicluster-data/master-2/wal/instance
I20260619 13:19:05.792390 388 external_mini_cluster.cc:1428] Setting key 9ec1460f20c0df493a3d853775e3ce8f
I20260619 13:19:05.796173 22959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:19:05 dist-test-slave-mkt3 krb5kdc[22731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 13:19:05 dist-test-slave-mkt3 krb5kdc[22731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:19:05.813540 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.799277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.0.97.62:44877)
0619 13:19:05.799426 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:05.799430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:05.800036 (+ 606us) server_negotiation.cc:407] Connection header received
0619 13:19:05.800194 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:05.800199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:05.800275 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:05.800354 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:05.801684 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.802440 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.804033 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.804440 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.804533 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.808016 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:05.808021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:05.808024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:05.808053 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:05.810731 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.811499 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.811504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.811507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.811662 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.812019 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.812022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.812024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.812307 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:05.812417 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:05.813041 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 13:19:05.813265 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 13:19:05.814709 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.799258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44877 (local address 127.0.0.1:41642)
0619 13:19:05.799907 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.799945 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:05.800080 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:05.800437 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:05.800444 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:05.801220 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:05.801541 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:05.801554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.802604 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.802608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:05.803884 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:05.803899 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.804665 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.804674 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:05.804867 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.805421 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:05.805449 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:05.807833 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:05.810869 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:05.810878 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:05.810888 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:05.811390 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:05.811754 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:05.811758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:05.811760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:05.811926 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:05.812421 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:05.812429 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:05.812665 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 13:19:05.812909 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 13:19:05.813415 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":130,"thread_start_us":348,"threads_started":1}
I20260619 13:19:05.814726 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.794046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.0.97.60:41767)
0619 13:19:05.794739 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:05.794754 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:05.794798 (+ 44us) server_negotiation.cc:407] Connection header received
0619 13:19:05.796790 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:05.796817 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:05.797315 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:05.797719 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:05.798960 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.800773 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.802320 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.803061 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.803194 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.807556 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:05.807568 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:05.807583 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:05.807631 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:05.810363 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.811130 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.811138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.811142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.811321 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.811727 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.811732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.811735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.812349 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:05.812647 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:05.812881 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 13:19:05.813304 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":470,"thread_start_us":269,"threads_started":1}
I20260619 13:19:05.817114 22959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:19:05 dist-test-slave-mkt3 krb5kdc[22731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 13:19:05 dist-test-slave-mkt3 krb5kdc[22731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:19:05.829985 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.817657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43615 (local address 127.0.0.1:34742)
0619 13:19:05.817796 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.817811 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:05.817918 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:05.818345 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:05.818349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:05.818685 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:05.818820 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:05.818828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.819863 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.819867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:05.820708 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:05.820718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.821408 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.821414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:05.821599 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.822120 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:05.822136 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:05.824389 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:05.827605 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:05.827610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:05.827614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:05.828061 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:05.828410 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:05.828415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:05.828419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:05.828578 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:05.829142 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:05.829146 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:05.829297 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 13:19:05.829429 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 13:19:05.829624 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 13:19:05.830058 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.817889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.0.97.61:43615)
0619 13:19:05.818071 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:05.818076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:05.818094 (+ 18us) server_negotiation.cc:407] Connection header received
0619 13:19:05.818149 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:05.818153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:05.818225 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:05.818310 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:05.818951 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.819731 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.820832 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.821281 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.821387 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.824589 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:05.824596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:05.824601 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:05.824639 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:05.827482 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.828191 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.828195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.828197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.828316 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.828684 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.828688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.828692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.829014 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:05.829163 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:05.829547 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 13:19:05.829746 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 13:19:05.830914 22959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:19:05.831341 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.817567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41648 (local address 127.0.97.62:44877)
0619 13:19:05.818066 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:05.818072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:05.818087 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:19:05.818128 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:05.818131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:05.818181 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:05.818260 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:05.819344 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.820111 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.821260 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.821671 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.821772 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.825916 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:05.825921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:05.825924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:05.825952 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:05.828475 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.829129 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.829134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.829137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.829275 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.829660 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.829665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.829668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.830007 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:05.830263 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:05.830780 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 13:19:05.830986 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392}
Jun 19 13:19:05 dist-test-slave-mkt3 krb5kdc[22731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 13:19:05 dist-test-slave-mkt3 krb5kdc[22731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:19:05.843638 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.831954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.0.97.60:41767)
0619 13:19:05.832074 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:05.832097 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:05.832110 (+ 13us) server_negotiation.cc:407] Connection header received
0619 13:19:05.832145 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:05.832148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:05.832208 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:05.832288 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:05.833064 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.833920 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.835248 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.835646 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.835746 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.838626 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:05.838632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:05.838636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:05.838669 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:05.840916 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.841509 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.841515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.841519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.841660 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.842069 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.842075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.842078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.842395 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:05.842529 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:05.842917 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 13:19:05.843300 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":38}
I20260619 13:19:05.843712 22963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.831666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46538)
0619 13:19:05.831853 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.831870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:05.831965 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:05.832348 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:05.832352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:05.832731 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:05.832893 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:05.832904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.834046 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.834051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:05.835132 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:05.835143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.835764 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.835770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:05.835960 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.836423 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:05.836439 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:05.838468 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:05.841035 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:05.841040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:05.841044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:05.841391 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:05.841777 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:05.841782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:05.841786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:05.841956 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:05.842523 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:05.842527 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:05.842679 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 13:19:05.842830 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 13:19:05.843121 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 13:19:05.848182 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.832878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34754 (local address 127.0.97.61:43615)
0619 13:19:05.833014 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:05.833020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:05.833066 (+ 46us) server_negotiation.cc:407] Connection header received
0619 13:19:05.833217 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:05.833222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:05.833278 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:05.833351 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:05.834739 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.835764 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.837428 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.838058 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.838186 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.842767 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:05.842773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:05.842778 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:05.842812 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:05.845575 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.846246 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.846251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.846255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.846395 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.846870 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.846875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.846878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.847205 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:05.847446 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:05.847618 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 13:19:05.847830 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:19:05.849231 22959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f: Bootstrap starting.
I20260619 13:19:05.852615 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:19:05.852924 22959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f: Neither blocks nor log segments found. Creating new log.
I20260619 13:19:05.854166 22959 log.cc:824] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f: Log is configured to *not* fsync() on all Append() calls
I20260619 13:19:05.856752 22959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f: No bootstrap required, opened a new log
I20260619 13:19:05.855751 22786 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41648
I20260619 13:19:05.863408 22959 raft_consensus.cc:348] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } }
I20260619 13:19:05.863699 22959 raft_consensus.cc:729] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c16583fa101a47a6a483ed5cc92e828f, State: Initialized, Role: FOLLOWER
I20260619 13:19:05.864090 22959 consensus_queue.cc:260] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } }
I20260619 13:19:05.864912 22965 sys_catalog.cc:455] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } } }
I20260619 13:19:05.865289 22965 sys_catalog.cc:458] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:19:05.865728 22959 sys_catalog.cc:564] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f [sys.catalog]: configured and running, proceeding with master startup.
W20260619 13:19:05.874235 22976 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:19:05.874362 22976 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:19:05.872066 22892 master_runner.cc:186] Error getting master registration for 127.0.97.62:44877: OK, instance_id { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" instance_seqno: 1781875144829510 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:19:05.879652 22892 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:19:05.960484 22811 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } attempt: 4
Jun 19 13:19:05 dist-test-slave-mkt3 krb5kdc[22731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 13:19:05.974862 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.961280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46546)
0619 13:19:05.961559 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:05.961580 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:05.961707 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:05.962090 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:05.962095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:05.962485 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:05.962650 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:05.962658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.963685 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.963691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:05.964633 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:05.964646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.965357 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.965363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:05.965585 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.966243 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:05.966262 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:05.968788 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:05.972120 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:05.972125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:05.972130 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:05.972629 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:05.973009 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:05.973014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:05.973018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:05.973205 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:05.973858 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:05.973864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:05.974029 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 13:19:05.974177 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 13:19:05.974461 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260619 13:19:05.974900 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:05.961365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46546 (local address 127.0.97.60:41767)
0619 13:19:05.961600 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:05.961606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:05.961629 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:19:05.961868 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:05.961872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:05.961938 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:05.962032 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:05.962797 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.963540 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.964769 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:05.965221 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:05.965327 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:05.968969 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:05.968975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:05.968978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:05.969012 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:05.971992 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.972738 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.972743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.972747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.972904 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:05.973309 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:05.973314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:05.973318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:05.973722 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:05.973910 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:05.974293 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 13:19:05.974525 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 13:19:05.979566 22811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1: Bootstrap starting.
I20260619 13:19:05.983666 22811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1: Neither blocks nor log segments found. Creating new log.
I20260619 13:19:05.984747 22811 log.cc:824] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1: Log is configured to *not* fsync() on all Append() calls
I20260619 13:19:05.987455 22811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1: No bootstrap required, opened a new log
I20260619 13:19:05.992686 22811 raft_consensus.cc:348] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } }
I20260619 13:19:05.992990 22811 raft_consensus.cc:729] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94ff5be567f47e2b516abcf5fbb7bb1, State: Initialized, Role: FOLLOWER
I20260619 13:19:05.993420 22811 consensus_queue.cc:260] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } }
I20260619 13:19:05.994307 22979 sys_catalog.cc:455] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } } }
I20260619 13:19:05.994673 22979 sys_catalog.cc:458] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:19:05.995446 22811 sys_catalog.cc:564] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:19:05.995505 22786 raft_consensus.cc:482] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 13:19:05.995723 22786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:19:05.999069 22786 raft_consensus.cc:504] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } }
I20260619 13:19:06.001183 22786 leader_election.cc:288] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [CANDIDATE]: Term 1 election: Requested vote from peers 4783b70262834e15a4ce574bd9bdce00 (127.0.97.61:43615), c16583fa101a47a6a483ed5cc92e828f (127.0.97.60:41767)
I20260619 13:19:06.001515 22934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c16583fa101a47a6a483ed5cc92e828f"
I20260619 13:19:06.002043 22934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:19:06.005470 22934 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94ff5be567f47e2b516abcf5fbb7bb1 in term 1.
I20260619 13:19:06.006120 22763 leader_election.cc:302] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b94ff5be567f47e2b516abcf5fbb7bb1, c16583fa101a47a6a483ed5cc92e828f; no voters:
I20260619 13:19:06.006587 22979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:19:06.007319 22979 raft_consensus.cc:686] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [term 1 LEADER]: Becoming Leader. State: Replica: b94ff5be567f47e2b516abcf5fbb7bb1, State: Running, Role: LEADER
I20260619 13:19:06.007889 22979 consensus_queue.cc:237] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } }
I20260619 13:19:06.015636 22984 sys_catalog.cc:455] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b94ff5be567f47e2b516abcf5fbb7bb1. Latest consensus state: current_term: 1 leader_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } } }
I20260619 13:19:06.016121 22984 sys_catalog.cc:458] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [sys.catalog]: This master's current role is: LEADER
I20260619 13:19:06.017287 22987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260619 13:19:06.019167 22913 tablet.cc:2369] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:19:06.035035 22885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } attempt: 5
Jun 19 13:19:06 dist-test-slave-mkt3 krb5kdc[22731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 13:19:06.057350 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:06.035637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41767 (local address 127.0.0.1:46560)
0619 13:19:06.036005 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:06.036025 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:06.036154 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:06.036664 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:06.036669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:06.037121 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:06.037327 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:06.037340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.038920 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.038926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:06.040300 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:06.040316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.041572 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.041586 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:06.041803 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:06.042643 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:06.042691 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:06.046474 (+ 3783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:06.052591 (+ 6117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:06.052597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:06.052621 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:06.053483 (+ 862us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:06.054156 (+ 673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:06.054161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:06.054165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:06.054421 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:06.055389 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:06.055394 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:06.055642 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 13:19:06.055841 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 13:19:06.056236 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260619 13:19:06.035787 22987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:19:06.063616 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:06.035976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.0.97.60:41767)
0619 13:19:06.036162 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.036167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.036186 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:19:06.036254 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.036259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.036351 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.036446 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.037544 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.038714 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.040493 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.041225 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.041348 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:06.046754 (+ 5406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:06.046761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:06.046766 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:06.046808 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:06.052383 (+ 5575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:06.053702 (+ 1319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:06.053732 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:06.053736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:06.053967 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:06.054638 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:06.054643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:06.054670 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:06.055211 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:06.056576 (+ 1365us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:06.057149 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 13:19:06.058340 (+ 1191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 13:19:06.069685 22885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00: Bootstrap starting.
I20260619 13:19:06.073684 22885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00: Neither blocks nor log segments found. Creating new log.
W20260619 13:19:06.074319 22765 tablet.cc:2369] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 13:19:06.076045 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:06.035152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.0.97.62:44877)
0619 13:19:06.035317 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.035323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.039177 (+ 3854us) server_negotiation.cc:407] Connection header received
0619 13:19:06.040033 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.040039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.040122 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.040191 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.044708 (+ 4517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.045805 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.049437 (+ 3632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.050150 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.050244 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:06.057020 (+ 6776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:06.057027 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:06.057031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:06.057073 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:06.062245 (+ 5172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:06.062944 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:06.062949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:06.062952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:06.063131 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:06.063602 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:06.063608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:06.063611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:06.074421 (+ 10810us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:06.074703 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:06.075433 (+ 730us) server_negotiation.cc:299] Negotiation successful
0619 13:19:06.075696 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 13:19:06.084457 22885 log.cc:824] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00: Log is configured to *not* fsync() on all Append() calls
I20260619 13:19:06.085122 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:06.044394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.0.97.61:43615)
0619 13:19:06.044568 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.044574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.053763 (+ 9189us) server_negotiation.cc:407] Connection header received
0619 13:19:06.056164 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.056170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.056267 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.056337 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.061455 (+ 5118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.062621 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.068009 (+ 5388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.068697 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.068795 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:06.075445 (+ 6650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:06.075450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:06.075454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:06.075488 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:06.079527 (+ 4039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:06.082626 (+ 3099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:06.082633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:06.082636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:06.082811 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:06.083260 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:06.083266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:06.083269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:06.083638 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:06.083897 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:06.084503 (+ 606us) server_negotiation.cc:299] Negotiation successful
0619 13:19:06.084762 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 13:19:06.090519 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:06.038967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46572 (local address 127.0.97.60:41767)
0619 13:19:06.059252 (+ 20285us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.059258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.059283 (+ 25us) server_negotiation.cc:407] Connection header received
0619 13:19:06.059336 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.059342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.059410 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.059508 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.064502 (+ 4994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.065942 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.069739 (+ 3797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.070516 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.070647 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:06.076720 (+ 6073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:06.076735 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:06.076742 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:06.076784 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:06.081174 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:06.087198 (+ 6024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:06.087205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:06.087209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:06.087373 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:06.087810 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:06.087881 (+ 71us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:06.087885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:06.088345 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:06.088586 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:06.089773 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0619 13:19:06.090071 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20146,"thread_start_us":22305,"threads_started":1}
I20260619 13:19:06.099938 22885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00: No bootstrap required, opened a new log
I20260619 13:19:06.110524 22885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } }
I20260619 13:19:06.111151 22885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4783b70262834e15a4ce574bd9bdce00, State: Initialized, Role: FOLLOWER
I20260619 13:19:06.111874 22885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } }
I20260619 13:19:06.112356 22934 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f [term 1 FOLLOWER]: Refusing update from remote peer b94ff5be567f47e2b516abcf5fbb7bb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:19:06.113291 22984 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:19:06.100867 22744 master_runner.cc:186] Error getting master registration for 127.0.97.61:43615: OK, instance_id { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" instance_seqno: 1781875145316338 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:19:06.118187 22744 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:19:06.122440 23005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } } }
I20260619 13:19:06.122881 23005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:19:06.123651 22885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:19:06.123782 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4783b70262834e15a4ce574bd9bdce00"
I20260619 13:19:06.124217 22860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:19:06.127840 22860 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b94ff5be567f47e2b516abcf5fbb7bb1 in term 1.
I20260619 13:19:06.129467 22859 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00 [term 1 FOLLOWER]: Refusing update from remote peer b94ff5be567f47e2b516abcf5fbb7bb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:19:06.130425 22984 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:19:06.144315 22965 sys_catalog.cc:455] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b94ff5be567f47e2b516abcf5fbb7bb1. Latest consensus state: current_term: 1 leader_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } } }
I20260619 13:19:06.144598 22965 sys_catalog.cc:458] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:19:06.146772 23004 mvcc.cc:205] Tried to move back new op lower bound from 7298560598466953216 to 7298560598053605376. Current Snapshot: MvccSnapshot[applied={T|T < 7298560598466953216}]
I20260619 13:19:06.156745 22965 sys_catalog.cc:455] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } } }
I20260619 13:19:06.157011 22965 sys_catalog.cc:458] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:19:06.167248 22984 sys_catalog.cc:455] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } } }
I20260619 13:19:06.167575 22984 sys_catalog.cc:458] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [sys.catalog]: This master's current role is: LEADER
I20260619 13:19:06.169970 22987 catalog_manager.cc:1347] Generated new cluster ID: 99a98f52e6694735bc2d3effc9b7a8d2
I20260619 13:19:06.170081 22987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:19:06.178779 23005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b94ff5be567f47e2b516abcf5fbb7bb1. Latest consensus state: current_term: 1 leader_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } } }
I20260619 13:19:06.179085 23005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:19:06.189795 22979 sys_catalog.cc:455] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } } }
I20260619 13:19:06.190157 22979 sys_catalog.cc:458] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1 [sys.catalog]: This master's current role is: LEADER
I20260619 13:19:06.190619 23005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94ff5be567f47e2b516abcf5fbb7bb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44877 } } peers { permanent_uuid: "4783b70262834e15a4ce574bd9bdce00" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43615 } } peers { permanent_uuid: "c16583fa101a47a6a483ed5cc92e828f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41767 } } }
I20260619 13:19:06.190874 23005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:19:06.224922 22987 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:19:06.226977 22987 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:19:06.250905 22987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b94ff5be567f47e2b516abcf5fbb7bb1: Generated new TSK 0
I20260619 13:19:06.252221 22987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:19:06.285645 23024 catalog_manager.cc:1259] Loaded cluster ID: 99a98f52e6694735bc2d3effc9b7a8d2
I20260619 13:19:06.285755 23024 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00: loading cluster ID for follower catalog manager: success
I20260619 13:19:06.289624 23024 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00: acquiring CA information for follower catalog manager: success
I20260619 13:19:06.292027 23024 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4783b70262834e15a4ce574bd9bdce00: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 13:19:06.384800 22813 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:06.368462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.0.97.62:44877)
0619 13:19:06.368668 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.368674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.382422 (+ 13748us) server_negotiation.cc:407] Connection header received
0619 13:19:06.383253 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.383261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.383363 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.383445 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.384556 (+ 1111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41674: BlockingRecv error: recv got EOF from 127.0.0.1:41674 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 13:19:06.386961 22996 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:06.368484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.0.97.60:41767)
0619 13:19:06.368683 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.368688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.382962 (+ 14274us) server_negotiation.cc:407] Connection header received
0619 13:19:06.385438 (+ 2476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.385443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.385533 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.385603 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.386741 (+ 1138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46574: BlockingRecv error: recv got EOF from 127.0.0.1:46574 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 13:19:06.389483 22889 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:06.382728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34770 (local address 127.0.97.61:43615)
0619 13:19:06.382937 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.382944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.387674 (+ 4730us) server_negotiation.cc:407] Connection header received
0619 13:19:06.388222 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.388227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.388338 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.388408 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.389270 (+ 862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34770: BlockingRecv error: recv got EOF from 127.0.0.1:34770 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:19:06.403398 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:06.395925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34784 (local address 127.0.97.61:43615)
0619 13:19:06.396185 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.396190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.396238 (+ 48us) server_negotiation.cc:407] Connection header received
0619 13:19:06.396455 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.396460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.396686 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.396837 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:06.397272 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.398474 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.400410 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.401139 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.401343 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:06.401911 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:06.402591 (+ 680us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:06.402735 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:06.402869 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 13:19:06.402954 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
W20260619 13:19:06.415863 22996 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:06.410091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46586 (local address 127.0.97.60:41767)
0619 13:19:06.410263 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.410269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.414367 (+ 4098us) server_negotiation.cc:407] Connection header received
0619 13:19:06.414587 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.414593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.414730 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.414834 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.415624 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46586: BlockingRecv error: recv got EOF from 127.0.0.1:46586 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:19:06.429219 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:06.422377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34800 (local address 127.0.97.61:43615)
0619 13:19:06.422571 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.422577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.423050 (+ 473us) server_negotiation.cc:407] Connection header received
0619 13:19:06.423238 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.423244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.423348 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.423435 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:06.423894 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.425069 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.427096 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.427856 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.427988 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:06.428430 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:06.428600 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:06.428690 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:06.428823 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 13:19:06.428911 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 13:19:06.438503 22996 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:06.436588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.0.97.60:41767)
0619 13:19:06.436764 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.436769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.437208 (+ 439us) server_negotiation.cc:407] Connection header received
0619 13:19:06.437361 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.437366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.437453 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.437550 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.438299 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46596: BlockingRecv error: recv got EOF from 127.0.0.1:46596 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 13:19:06.453981 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:06.447431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.0.97.61:43615)
0619 13:19:06.447594 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.447599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.448039 (+ 440us) server_negotiation.cc:407] Connection header received
0619 13:19:06.448192 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.448197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.448295 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.448384 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:06.448768 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.449976 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.451782 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.452500 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.452640 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:06.453043 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:06.453224 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:06.453311 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:06.453571 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 13:19:06.453651 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 13:19:06.462841 22996 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:06.460677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.0.97.60:41767)
0619 13:19:06.460835 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.460841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.461532 (+ 691us) server_negotiation.cc:407] Connection header received
0619 13:19:06.461696 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.461701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.461791 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.461921 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.462637 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46612: BlockingRecv error: recv got EOF from 127.0.0.1:46612 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 13:19:06.476586 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:06.470158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.0.97.61:43615)
0619 13:19:06.470359 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.470364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.470835 (+ 471us) server_negotiation.cc:407] Connection header received
0619 13:19:06.470979 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.470984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.471061 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.471166 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:06.471545 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.472662 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.474485 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.475223 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.475343 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:06.475881 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:06.476014 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:06.476112 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:06.476243 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 13:19:06.476323 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260619 13:19:06.485145 22996 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:06.483105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.0.97.60:41767)
0619 13:19:06.483326 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.483332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.483827 (+ 495us) server_negotiation.cc:407] Connection header received
0619 13:19:06.483986 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.483991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.484088 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.484186 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.484936 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46624: BlockingRecv error: recv got EOF from 127.0.0.1:46624 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260619 13:19:06.505170 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:06.498284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34842 (local address 127.0.97.61:43615)
0619 13:19:06.498463 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.498469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.498956 (+ 487us) server_negotiation.cc:407] Connection header received
0619 13:19:06.499142 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.499147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.499233 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.499334 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:06.499789 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.501008 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.503090 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.503779 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.503904 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:06.504401 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:06.504551 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:06.504637 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:06.504773 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 13:19:06.504854 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 13:19:06.513890 22996 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:06.511601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46640 (local address 127.0.97.60:41767)
0619 13:19:06.511771 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.511777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.512475 (+ 698us) server_negotiation.cc:407] Connection header received
0619 13:19:06.512665 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.512671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.512761 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.512852 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.513643 (+ 791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46640: BlockingRecv error: recv got EOF from 127.0.0.1:46640 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:19:06.540194 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:06.533413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34846 (local address 127.0.97.61:43615)
0619 13:19:06.533570 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.533576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.534082 (+ 506us) server_negotiation.cc:407] Connection header received
0619 13:19:06.534250 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.534256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.534341 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.534434 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:06.534867 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.536072 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.538029 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.538684 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.538800 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:06.539384 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:06.539546 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:06.539642 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:06.539820 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 13:19:06.539907 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 13:19:06.554188 22996 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:06.551902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46654 (local address 127.0.97.60:41767)
0619 13:19:06.552124 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.552130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.552609 (+ 479us) server_negotiation.cc:407] Connection header received
0619 13:19:06.552791 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.552797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.552894 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.553056 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.553949 (+ 893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46654: BlockingRecv error: recv got EOF from 127.0.0.1:46654 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 13:19:06.604878 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:06.591417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.0.97.61:43615)
0619 13:19:06.591638 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.591645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.592174 (+ 529us) server_negotiation.cc:407] Connection header received
0619 13:19:06.592534 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.592540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.592654 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.592764 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:06.593235 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.594419 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.596407 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.597428 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.597584 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:06.601338 (+ 3754us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:06.601491 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:06.601584 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:06.604427 (+ 2843us) server_negotiation.cc:299] Negotiation successful
0619 13:19:06.604520 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 13:19:06.614239 22996 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:06.611922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46662 (local address 127.0.97.60:41767)
0619 13:19:06.612097 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.612103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.612823 (+ 720us) server_negotiation.cc:407] Connection header received
0619 13:19:06.612991 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.612996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.613086 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.613181 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.613988 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46662: BlockingRecv error: recv got EOF from 127.0.0.1:46662 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 13:19:06.687881 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:06.682274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.0.97.61:43615)
0619 13:19:06.682468 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.682473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.682839 (+ 366us) server_negotiation.cc:407] Connection header received
0619 13:19:06.683025 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.683029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.683096 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.683188 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:06.683637 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.684349 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.686206 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.686679 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.686802 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:06.687245 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:06.687350 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:06.687411 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:06.687568 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 13:19:06.687631 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 13:19:06.695526 22996 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:06.693734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46670 (local address 127.0.97.60:41767)
0619 13:19:06.693939 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.693945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.694211 (+ 266us) server_negotiation.cc:407] Connection header received
0619 13:19:06.694353 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.694359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.694456 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.694562 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.695323 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46670: BlockingRecv error: recv got EOF from 127.0.0.1:46670 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:19:06.831627 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:06.826966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34872 (local address 127.0.97.61:43615)
0619 13:19:06.827170 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.827177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.827477 (+ 300us) server_negotiation.cc:407] Connection header received
0619 13:19:06.827633 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.827637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.827707 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.827797 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:06.828154 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.828845 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.830184 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:06.830591 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:06.830696 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:06.831031 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:06.831136 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:06.831201 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:06.831291 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 13:19:06.831353 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 13:19:06.843142 22996 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:06.841348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46686 (local address 127.0.97.60:41767)
0619 13:19:06.841547 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:06.841552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:06.841809 (+ 257us) server_negotiation.cc:407] Connection header received
0619 13:19:06.842000 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:06.842004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:06.842077 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:06.842158 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:06.842925 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46686: BlockingRecv error: recv got EOF from 127.0.0.1:46686 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:19:06.876293 22976 catalog_manager.cc:1259] Loaded cluster ID: 99a98f52e6694735bc2d3effc9b7a8d2
I20260619 13:19:06.876390 22976 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f: loading cluster ID for follower catalog manager: success
I20260619 13:19:06.878907 22976 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f: acquiring CA information for follower catalog manager: success
I20260619 13:19:06.880622 22976 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c16583fa101a47a6a483ed5cc92e828f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:19:07.108290 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:07.103285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34880 (local address 127.0.97.61:43615)
0619 13:19:07.103509 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:07.103515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:07.103775 (+ 260us) server_negotiation.cc:407] Connection header received
0619 13:19:07.103914 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:07.103918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:07.104003 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:07.104096 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:07.104486 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:07.105230 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:07.106549 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:07.106960 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:07.107054 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:07.107561 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:07.107708 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:07.107779 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:07.107944 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 13:19:07.108020 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 13:19:07.119059 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:07.113907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46702 (local address 127.0.97.60:41767)
0619 13:19:07.114073 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:07.114077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:07.114210 (+ 133us) server_negotiation.cc:407] Connection header received
0619 13:19:07.114323 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:07.114327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:07.114480 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:07.114578 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:07.115009 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:07.115708 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:07.117001 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:07.117395 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:07.117484 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:07.118027 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:07.118565 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:07.118677 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:07.118781 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 13:19:07.118851 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 13:19:07.132453 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:07.124929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41682 (local address 127.0.97.62:44877)
0619 13:19:07.125996 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:07.126002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:07.126021 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:19:07.126138 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:07.126144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:07.126407 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:07.126584 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:07.127094 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:07.128031 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:07.129722 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:07.130425 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:07.130605 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:07.131121 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:07.131587 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:07.131706 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:07.132058 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 13:19:07.132149 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":826,"threads_started":1}
I20260619 13:19:07.132484 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:07.124902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.0.97.60:41767)
0619 13:19:07.125109 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:07.125115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:07.126470 (+ 1355us) server_negotiation.cc:407] Connection header received
0619 13:19:07.126740 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:07.126745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:07.126805 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:07.126856 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:07.127591 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:07.128632 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:07.130542 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:07.131102 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:07.131206 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:07.131582 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:07.131700 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:07.131786 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:07.132154 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 13:19:07.132229 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:19:07.132540 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:07.125510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34884 (local address 127.0.97.61:43615)
0619 13:19:07.125679 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:07.125684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:07.126674 (+ 990us) server_negotiation.cc:407] Connection header received
0619 13:19:07.126955 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:07.126961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:07.127045 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:07.127132 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:07.127748 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:07.128770 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:07.130714 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:07.131208 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:07.131298 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:07.131834 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:07.131931 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:07.132004 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:07.132174 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 13:19:07.132255 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 13:19:07.142791 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 22744
I20260619 13:19:07.143922 22806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:19:07.251302 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 22744
I20260619 13:19:07.268627 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 22818
I20260619 13:19:07.269680 22880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:19:07.374050 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 22818
I20260619 13:19:07.390218 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 22892
I20260619 13:19:07.391322 22954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:19:07.496367 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 22892
2026-06-19T13:19:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4305 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 13:19:07 dist-test-slave-mkt3 krb5kdc[23149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 13:19:07 dist-test-slave-mkt3 krb5kdc[23149](info): set up 2 sockets
Jun 19 13:19:07 dist-test-slave-mkt3 krb5kdc[23149](info): commencing operation
krb5kdc: starting...
W20260619 13:19:08.502112 388 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/test-user.keytab.
Jun 19 13:19:08 dist-test-slave-mkt3 krb5kdc[23149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T13:19:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T13:19:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 13:19:08.726223 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38913
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38913,127.0.97.61:41423,127.0.97.60:36535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 13:19:09.040730 23162 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:19:09.041071 23162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:19:09.041175 23162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:19:09.041293 23162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:19:09.048802 23162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:19:09.048884 23162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:19:09.048966 23162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:19:09.049027 23162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:19:09.049110 23162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:19:09.058537 23162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38913,127.0.97.61:41423,127.0.97.60:36535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:19:09.059758 23162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:19:09.061035 23162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:19:09.072065 23167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:09.072124 23168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:09.072573 23170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:19:09.072768 23162 server_base.cc:1034] running on GCE node
I20260619 13:19:09.073554 23162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:19:09.074887 23162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:19:09.076102 23162 hybrid_clock.cc:648] HybridClock initialized: now 1781875149076036 us; error 42 us; skew 500 ppm
I20260619 13:19:09.076421 23162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:19:09 dist-test-slave-mkt3 krb5kdc[23149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:19:09.079382 23162 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 13:19:09.080674 23162 webserver.cc:460] Webserver started at http://127.0.97.62:44837/ using document root <none> and password file <none>
I20260619 13:19:09.081339 23162 fs_manager.cc:359] Metadata directory not provided
I20260619 13:19:09.081434 23162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:19:09.081713 23162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:19:09.084553 23162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/data/instance:
uuid: "5716d3f17c91418c817a7a68ede34959"
format_stamp: "Formatted at 2026-06-19 13:19:09 on dist-test-slave-mkt3"
server_key: "7fbd8d2516a7e006114c6d9c112a93ad"
server_key_iv: "77b818f92828aff5b78b72cb6564e573"
server_key_version: "encryptionkey@0"
I20260619 13:19:09.085325 23162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/wal/instance:
uuid: "5716d3f17c91418c817a7a68ede34959"
format_stamp: "Formatted at 2026-06-19 13:19:09 on dist-test-slave-mkt3"
server_key: "7fbd8d2516a7e006114c6d9c112a93ad"
server_key_iv: "77b818f92828aff5b78b72cb6564e573"
server_key_version: "encryptionkey@0"
I20260619 13:19:09.090150 23162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 13:19:09.093272 23177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:19:09.094514 23162 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 13:19:09.094668 23162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/wal
uuid: "5716d3f17c91418c817a7a68ede34959"
format_stamp: "Formatted at 2026-06-19 13:19:09 on dist-test-slave-mkt3"
server_key: "7fbd8d2516a7e006114c6d9c112a93ad"
server_key_iv: "77b818f92828aff5b78b72cb6564e573"
server_key_version: "encryptionkey@0"
I20260619 13:19:09.094853 23162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:19:09.110445 23162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:19:09.114636 23162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:19:09.114881 23162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:19:09.131996 23162 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38913
I20260619 13:19:09.132016 23228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38913 every 8 connection(s)
I20260619 13:19:09.133285 23162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/data/info.pb
I20260619 13:19:09.136812 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 23162
I20260619 13:19:09.137005 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-0/wal/instance
I20260619 13:19:09.137086 23229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:19:09.137573 388 external_mini_cluster.cc:1428] Setting key 5597a70f3c8dca2c3b6647b63b00b987
I20260619 13:19:09.145592 23229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:19:09 dist-test-slave-mkt3 krb5kdc[23149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260619 13:19:09.167831 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.149349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.0.97.62:38913)
0619 13:19:09.150184 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:09.150202 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:09.150608 (+ 406us) server_negotiation.cc:407] Connection header received
0619 13:19:09.152552 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:09.152558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:09.152809 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:09.152987 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:09.154323 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.155483 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.156823 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.157374 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.157542 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:09.161045 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:09.161053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:09.161064 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:09.161092 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:09.163874 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:09.164639 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:09.164646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:09.164648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:09.164796 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:09.165226 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:09.165230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:09.165232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:09.165707 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:09.165886 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:09.166502 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 13:19:09.166763 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":350,"threads_started":1}
I20260619 13:19:09.167856 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.149350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38913 (local address 127.0.0.1:53118)
0619 13:19:09.150452 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.150486 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:09.152184 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:09.153029 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:09.153037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:09.153727 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:09.154113 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:09.154133 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.155624 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.155627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:09.156696 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:09.156708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.157598 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.157605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:09.157776 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:09.158445 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:09.158466 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:09.160830 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:09.164030 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:09.164043 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:09.164055 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:09.164487 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:09.164908 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:09.164913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:09.164916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:09.165123 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:09.165833 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:09.165859 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:09.166105 (+ 246us) client_negotiation.cc:769] Sending connection context
0619 13:19:09.166356 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 13:19:09.166940 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":100,"thread_start_us":687,"threads_started":1}
I20260619 13:19:09.170737 23229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:19:09.171756 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.171349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41423 (local address 127.0.0.1:45434)
0619 13:19:09.171506 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.171623 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 13:19:09.172006 23179 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
I20260619 13:19:09.173452 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.173102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41423 (local address 127.0.0.1:45440)
0619 13:19:09.173244 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.173317 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 13:19:09.173691 23229 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41423: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 13:19:09.190220 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41423
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38913,127.0.97.61:41423,127.0.97.60:36535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:19:09.223073 23229 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } attempt: 1
I20260619 13:19:09.224288 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.223785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41423 (local address 127.0.0.1:45456)
0619 13:19:09.223994 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.224107 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 13:19:09.225261 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.224886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41423 (local address 127.0.0.1:45468)
0619 13:19:09.225031 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.225110 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 13:19:09.225485 23229 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41423: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
I20260619 13:19:09.293699 23229 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } attempt: 2
I20260619 13:19:09.294965 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.294411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41423 (local address 127.0.0.1:45478)
0619 13:19:09.294681 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.294791 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 13:19:09.296008 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.295576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41423 (local address 127.0.0.1:45488)
0619 13:19:09.295745 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.295847 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 13:19:09.296229 23229 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41423: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
I20260619 13:19:09.387465 23229 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } attempt: 3
I20260619 13:19:09.388585 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.388071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41423 (local address 127.0.0.1:45494)
0619 13:19:09.388295 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.388408 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 13:19:09.389627 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.389228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41423 (local address 127.0.0.1:45502)
0619 13:19:09.389373 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.389467 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 13:19:09.389856 23229 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41423: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
W20260619 13:19:09.511416 23236 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:19:09.511731 23236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:19:09.511801 23236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:19:09.511883 23236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:19:09.518894 23236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:19:09.518978 23236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:19:09.519037 23236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:19:09.519085 23236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:19:09.519135 23236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:19:09.528510 23236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38913,127.0.97.61:41423,127.0.97.60:36535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:19:09.529649 23236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:19:09.530946 23236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 13:19:09.533102 23229 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } attempt: 4
I20260619 13:19:09.534303 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.533723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41423 (local address 127.0.0.1:45518)
0619 13:19:09.534037 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.534144 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260619 13:19:09.535377 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.534964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41423 (local address 127.0.0.1:45534)
0619 13:19:09.535139 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.535235 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 13:19:09.535604 23229 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41423: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41423: connect: Connection refused (error 111)
W20260619 13:19:09.542311 23241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:09.542271 23242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:09.542265 23244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:19:09.543030 23236 server_base.cc:1034] running on GCE node
I20260619 13:19:09.543737 23236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:19:09.544973 23236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:19:09.546213 23236 hybrid_clock.cc:648] HybridClock initialized: now 1781875149546146 us; error 39 us; skew 500 ppm
I20260619 13:19:09.546535 23236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 13:19:09 dist-test-slave-mkt3 krb5kdc[23149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:19:09.549371 23236 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 13:19:09.550746 23236 webserver.cc:460] Webserver started at http://127.0.97.61:44567/ using document root <none> and password file <none>
I20260619 13:19:09.551388 23236 fs_manager.cc:359] Metadata directory not provided
I20260619 13:19:09.551482 23236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:19:09.551779 23236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:19:09.554615 23236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/data/instance:
uuid: "9b704e0bbe2844ad8acd2a9989d219be"
format_stamp: "Formatted at 2026-06-19 13:19:09 on dist-test-slave-mkt3"
server_key: "722ec1cd9b639de77ac57ff3107afe29"
server_key_iv: "782cfde9f80302ff0d399153a3ad8061"
server_key_version: "encryptionkey@0"
I20260619 13:19:09.555335 23236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/wal/instance:
uuid: "9b704e0bbe2844ad8acd2a9989d219be"
format_stamp: "Formatted at 2026-06-19 13:19:09 on dist-test-slave-mkt3"
server_key: "722ec1cd9b639de77ac57ff3107afe29"
server_key_iv: "782cfde9f80302ff0d399153a3ad8061"
server_key_version: "encryptionkey@0"
I20260619 13:19:09.559995 23236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 13:19:09.563248 23251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:19:09.564464 23236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:19:09.564607 23236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/wal
uuid: "9b704e0bbe2844ad8acd2a9989d219be"
format_stamp: "Formatted at 2026-06-19 13:19:09 on dist-test-slave-mkt3"
server_key: "722ec1cd9b639de77ac57ff3107afe29"
server_key_iv: "782cfde9f80302ff0d399153a3ad8061"
server_key_version: "encryptionkey@0"
I20260619 13:19:09.564795 23236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:19:09.584967 23236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:19:09.588614 23236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:19:09.588856 23236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:19:09.605618 23236 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41423
I20260619 13:19:09.605655 23302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41423 every 8 connection(s)
I20260619 13:19:09.606907 23236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/data/info.pb
I20260619 13:19:09.610361 23303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:19:09.611619 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 23236
I20260619 13:19:09.611801 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-1/wal/instance
I20260619 13:19:09.612329 388 external_mini_cluster.cc:1428] Setting key 5804ebe7b149b7cd50ef55d93a50d403
I20260619 13:19:09.618409 23303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:19:09 dist-test-slave-mkt3 krb5kdc[23149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260619 13:19:09.639279 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.621330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.0.97.62:38913)
0619 13:19:09.621533 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:09.621539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:09.622679 (+ 1140us) server_negotiation.cc:407] Connection header received
0619 13:19:09.624995 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:09.625000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:09.625071 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:09.625167 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:09.626985 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.627661 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.629605 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.630037 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.630155 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:09.633761 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:09.633765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:09.633768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:09.633799 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:09.636254 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:09.637122 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:09.637126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:09.637128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:09.637248 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:09.637676 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:09.637679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:09.637681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:09.637974 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:09.638083 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:09.638873 (+ 790us) server_negotiation.cc:299] Negotiation successful
0619 13:19:09.639034 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 13:19:09.640133 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.621743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38913 (local address 127.0.0.1:53120)
0619 13:19:09.622510 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.622559 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:09.624699 (+ 2140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:09.625416 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:09.625426 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:09.626115 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:09.626824 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:09.626845 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.627798 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.627803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:09.629453 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:09.629468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.630246 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.630252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:09.630489 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:09.631113 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:09.631137 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:09.633547 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:09.636390 (+ 2843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:09.636399 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:09.636415 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:09.636995 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:09.637349 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:09.637354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:09.637357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:09.637569 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:09.638096 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:09.638104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:09.638499 (+ 395us) client_negotiation.cc:769] Sending connection context
0619 13:19:09.638799 (+ 300us) client_negotiation.cc:240] Negotiation successful
0619 13:19:09.639137 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":310,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260619 13:19:09.641804 23303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:19:09 dist-test-slave-mkt3 krb5kdc[23149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:19:09.655859 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.642447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41423 (local address 127.0.0.1:45544)
0619 13:19:09.642683 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.642702 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:09.642789 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:09.643568 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:09.643576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:09.643943 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:09.644108 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:09.644118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.645369 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.645374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:09.646547 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:09.646560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.647297 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.647302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:09.647467 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:09.647960 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:09.647979 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:09.649981 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:09.653430 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:09.653436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:09.653440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:09.653927 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:09.654317 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:09.654322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:09.654326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:09.654457 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:09.654982 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:09.654986 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:09.655133 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 13:19:09.655242 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 13:19:09.655484 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260619 13:19:09.655953 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.642640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.0.97.61:41423)
0619 13:19:09.643203 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:09.643209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:09.643232 (+ 23us) server_negotiation.cc:407] Connection header received
0619 13:19:09.643271 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:09.643276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:09.643427 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:09.643532 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:09.644258 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.645231 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.646669 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.647166 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.647256 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:09.650190 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:09.650203 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:09.650219 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:09.650252 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:09.653300 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:09.654079 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:09.654086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:09.654088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:09.654218 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:09.654555 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:09.654559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:09.654561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:09.654868 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:09.655057 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:09.655399 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 13:19:09.655603 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":300,"threads_started":1}
I20260619 13:19:09.657997 23303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:19:09.659066 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.658622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45158)
0619 13:19:09.658771 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.658907 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 13:19:09.659322 23254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
I20260619 13:19:09.660862 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.660506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45174)
0619 13:19:09.660690 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.660763 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 13:19:09.661082 23303 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36535: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 13:19:09.668767 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
/tmp/dist-test-taskwhPxxl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36535
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38913,127.0.97.61:41423,127.0.97.60:36535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 13:19:09.710356 23303 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } attempt: 1
I20260619 13:19:09.711390 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.710970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45186)
0619 13:19:09.711153 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.711241 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 13:19:09.712493 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.712134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45194)
0619 13:19:09.712301 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.712376 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 13:19:09.712687 23303 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36535: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
I20260619 13:19:09.780900 23303 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } attempt: 2
I20260619 13:19:09.782104 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.781601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45206)
0619 13:19:09.781798 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.781942 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 13:19:09.783056 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.782730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45208)
0619 13:19:09.782873 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.782939 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 13:19:09.783242 23303 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36535: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
I20260619 13:19:09.834853 23229 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } attempt: 5
Jun 19 13:19:09 dist-test-slave-mkt3 krb5kdc[23149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:19:09.848737 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.835645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.0.97.61:41423)
0619 13:19:09.835804 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:09.835809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:09.835846 (+ 37us) server_negotiation.cc:407] Connection header received
0619 13:19:09.836066 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:09.836071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:09.836149 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:09.836225 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:09.837233 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.837992 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.839767 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.840222 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.840325 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:09.843813 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:09.843818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:09.843821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:09.843853 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:09.846370 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:09.846973 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:09.846977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:09.846980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:09.847109 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:09.847451 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:09.847455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:09.847457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:09.847721 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:09.847851 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:09.848214 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 13:19:09.848418 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 13:19:09.848737 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.835499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41423 (local address 127.0.0.1:45548)
0619 13:19:09.835753 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.835777 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:09.835912 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:09.836313 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:09.836318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:09.836835 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:09.837071 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:09.837084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.838149 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.838155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:09.839576 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:09.839594 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:09.840394 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:09.840400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:09.840662 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:09.841274 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:09.841298 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:09.843636 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:09.846502 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:09.846507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:09.846512 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:09.846866 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:09.847209 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:09.847215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:09.847219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:09.847353 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:09.847829 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:09.847834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:09.847977 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 13:19:09.848091 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 13:19:09.848361 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 13:19:09.849687 23229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:19:09.850720 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.850342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45210)
0619 13:19:09.850499 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.850590 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 13:19:09.851655 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.851347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45224)
0619 13:19:09.851481 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.851555 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 13:19:09.851832 23229 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36535: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
I20260619 13:19:09.874436 23303 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } attempt: 3
I20260619 13:19:09.875530 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.875080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45238)
0619 13:19:09.875286 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.875374 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 13:19:09.876628 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.876167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45242)
0619 13:19:09.876387 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.876482 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260619 13:19:09.876809 23303 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36535: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
I20260619 13:19:09.903057 23229 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } attempt: 1
I20260619 13:19:09.904045 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.903622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45256)
0619 13:19:09.903802 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.903883 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 13:19:09.905089 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.904755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45264)
0619 13:19:09.904907 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.904972 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 13:19:09.905284 23229 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36535: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
I20260619 13:19:09.973486 23229 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } attempt: 2
I20260619 13:19:09.974702 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.974192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45280)
0619 13:19:09.974448 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.974549 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 13:19:09.975653 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:09.975331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45282)
0619 13:19:09.975475 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:09.975539 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 13:19:09.975852 23229 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36535: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
W20260619 13:19:09.981698 23310 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 13:19:09.982050 23310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 13:19:09.982141 23310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 13:19:09.982259 23310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 13:19:09.989312 23310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 13:19:09.989398 23310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 13:19:09.989483 23310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 13:19:09.989548 23310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 13:19:09.989629 23310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 13:19:09.999312 23310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38913,127.0.97.61:41423,127.0.97.60:36535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 12:07:30 UTC on ccycloud.quasar-fpypta.root.comops.site
build id 4455651
ASAN enabled
I20260619 13:19:10.000547 23310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 13:19:10.001828 23310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 13:19:10.012887 23318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:10.012967 23315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 13:19:10.012944 23316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 13:19:10.013828 23310 server_base.cc:1034] running on GCE node
I20260619 13:19:10.014602 23310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 13:19:10.016201 23310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 13:19:10.017426 23310 hybrid_clock.cc:648] HybridClock initialized: now 1781875150017340 us; error 55 us; skew 500 ppm
I20260619 13:19:10.017738 23310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 13:19:10.020025 23303 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } attempt: 4
Jun 19 13:19:10 dist-test-slave-mkt3 krb5kdc[23149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 13:19:10.020793 23310 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 13:19:10.021052 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.020579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45298)
0619 13:19:10.020790 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:10.020868 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 13:19:10.022168 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.021792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45310)
0619 13:19:10.021980 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:10.022052 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 13:19:10.022408 23310 webserver.cc:460] Webserver started at http://127.0.97.60:33389/ using document root <none> and password file <none>
W20260619 13:19:10.022398 23303 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36535: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36535: connect: Connection refused (error 111)
I20260619 13:19:10.023074 23310 fs_manager.cc:359] Metadata directory not provided
I20260619 13:19:10.023180 23310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 13:19:10.023474 23310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 13:19:10.026397 23310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/data/instance:
uuid: "c9c0b884473b4790a0351c6dd9e22425"
format_stamp: "Formatted at 2026-06-19 13:19:10 on dist-test-slave-mkt3"
server_key: "0e8b141c7eee7150652057eee3956f39"
server_key_iv: "47571a8564b1aef2b13308248b42105c"
server_key_version: "encryptionkey@0"
I20260619 13:19:10.027143 23310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/wal/instance:
uuid: "c9c0b884473b4790a0351c6dd9e22425"
format_stamp: "Formatted at 2026-06-19 13:19:10 on dist-test-slave-mkt3"
server_key: "0e8b141c7eee7150652057eee3956f39"
server_key_iv: "47571a8564b1aef2b13308248b42105c"
server_key_version: "encryptionkey@0"
I20260619 13:19:10.032173 23310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 13:19:10.035485 23325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 13:19:10.036746 23310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 13:19:10.036890 23310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/wal
uuid: "c9c0b884473b4790a0351c6dd9e22425"
format_stamp: "Formatted at 2026-06-19 13:19:10 on dist-test-slave-mkt3"
server_key: "0e8b141c7eee7150652057eee3956f39"
server_key_iv: "47571a8564b1aef2b13308248b42105c"
server_key_version: "encryptionkey@0"
I20260619 13:19:10.037076 23310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 13:19:10.054730 23310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 13:19:10.058447 23310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 13:19:10.058689 23310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 13:19:10.075321 23310 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36535
I20260619 13:19:10.075340 23376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36535 every 8 connection(s)
I20260619 13:19:10.076560 23310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/data/info.pb
I20260619 13:19:10.079802 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu as pid 23310
I20260619 13:19:10.079979 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwhPxxl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781874807295266-388-0/minicluster-data/master-2/wal/instance
I20260619 13:19:10.080212 23377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 13:19:10.080417 388 external_mini_cluster.cc:1428] Setting key 24a13e3654c45b7a4f0a7dc4c9bf4513
I20260619 13:19:10.082027 23229 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } attempt: 3
I20260619 13:19:10.090818 23377 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 13:19:10 dist-test-slave-mkt3 krb5kdc[23149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 13:19:10 dist-test-slave-mkt3 krb5kdc[23149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 13:19:10 dist-test-slave-mkt3 krb5kdc[23149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:19:10.105201 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.082655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45332)
0619 13:19:10.082959 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:10.082977 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:10.083087 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:10.088348 (+ 5261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:10.088353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:10.088882 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:10.089063 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:10.089075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.091293 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.091299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:10.092682 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:10.092698 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.094106 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.094112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:10.094296 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.094895 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:10.094917 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:10.097390 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:10.102265 (+ 4875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:10.102270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:10.102274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:10.102749 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:10.103177 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:10.103182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:10.103185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:10.103337 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:10.104160 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:10.104165 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:10.104326 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 13:19:10.104515 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 13:19:10.104782 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260619 13:19:10.106092 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.082838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.0.97.60:36535)
0619 13:19:10.086102 (+ 3264us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.086108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.086130 (+ 22us) server_negotiation.cc:407] Connection header received
0619 13:19:10.087513 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.087526 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.087942 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.089198 (+ 1256us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.089452 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.091150 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.092834 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.093939 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.094666 (+ 727us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.097693 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:10.097703 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:10.097719 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:10.097782 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:10.102108 (+ 4326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.102868 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.102872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.102875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.103052 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.103434 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.103440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.103443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.104020 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:10.104198 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:10.104699 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 13:19:10.104967 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":3126,"spinlock_wait_cycles":55808,"thread_start_us":2600,"threads_started":1}
I20260619 13:19:10.107117 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.082126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45326 (local address 127.0.97.60:36535)
0619 13:19:10.085890 (+ 3764us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.085909 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.085959 (+ 50us) server_negotiation.cc:407] Connection header received
0619 13:19:10.087502 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.087522 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.087939 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.088288 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.089619 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.091056 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.092788 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.093934 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.094675 (+ 741us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.097693 (+ 3018us) spinlock_profiling.cc:243] Waited 2.05 ms on lock 0x60c0000347b8. stack: 00007fa0bc33c66e 00007fa0bc33c430 00007fa0bb3c35ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fa0bc332824 00007fa0bc331d66 00007fa0c2e49664 00007fa0c2e63e8d 00007fa0c2e5ef76 00007fa0c2e588a5 00007fa0c2dc6ea5 00007fa0c2dc5eff 00007fa0c2df982f 00007fa0c2df958d
0619 13:19:10.098490 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:10.098495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:10.098499 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:10.098532 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:10.102172 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.102846 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.102854 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.102862 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.103052 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.103744 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.103749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.103752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.104038 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:10.104432 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:10.106563 (+ 2131us) server_negotiation.cc:299] Negotiation successful
0619 13:19:10.106760 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":3436,"spinlock_wait_cycles":4720512,"thread_start_us":3545,"threads_started":1}
I20260619 13:19:10.108592 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.093104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53124 (local address 127.0.97.62:38913)
0619 13:19:10.093274 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.093279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.094072 (+ 793us) server_negotiation.cc:407] Connection header received
0619 13:19:10.094317 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.094322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.094397 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.094505 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.095798 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.096847 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.098587 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.099092 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.099213 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.102715 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:10.102721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:10.102725 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:10.102760 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:10.105764 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.106579 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.106583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.106587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.106731 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.107146 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.107151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.107154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.107468 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:10.107599 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:10.108076 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 13:19:10.108270 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:19:10.109290 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.093209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38913 (local address 127.0.0.1:53124)
0619 13:19:10.093968 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:10.094007 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:10.094184 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:10.094576 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:10.094582 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:10.095358 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:10.095643 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:10.095654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.097062 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.097066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:10.098470 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:10.098482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.099484 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.099490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:10.099638 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.100182 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:10.100204 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:10.102515 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:10.105910 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:10.105918 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:10.105924 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:10.106427 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:10.106845 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:10.106850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:10.106857 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:10.107013 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:10.107605 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:10.107619 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:10.107768 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 13:19:10.108754 (+ 986us) client_negotiation.cc:240] Negotiation successful
0619 13:19:10.108983 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":376,"threads_started":1}
I20260619 13:19:10.110880 23377 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:19:10.113253 23229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959: Bootstrap starting.
I20260619 13:19:10.117000 23229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959: Neither blocks nor log segments found. Creating new log.
Jun 19 13:19:10 dist-test-slave-mkt3 krb5kdc[23149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 13:19:10.118351 23229 log.cc:824] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959: Log is configured to *not* fsync() on all Append() calls
Jun 19 13:19:10 dist-test-slave-mkt3 krb5kdc[23149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 13:19:10.121552 23229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959: No bootstrap required, opened a new log
I20260619 13:19:10.125171 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.111392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41423 (local address 127.0.0.1:45560)
0619 13:19:10.111532 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:10.111547 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:10.111616 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:10.114478 (+ 2862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:10.114482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:10.114825 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:10.114942 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:10.114950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.115965 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.115969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:10.116792 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:10.116802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.117495 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.117501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:10.117658 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.118128 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:10.118146 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:10.120146 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:10.122909 (+ 2763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:10.122914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:10.122918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:10.123262 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:10.123584 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:10.123589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:10.123593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:10.123749 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:10.124322 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:10.124326 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:10.124465 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 13:19:10.124604 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 13:19:10.124844 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 13:19:10.125267 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.114091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.0.97.61:41423)
0619 13:19:10.114241 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.114246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.114260 (+ 14us) server_negotiation.cc:407] Connection header received
0619 13:19:10.114303 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.114306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.114360 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.114441 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.115079 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.115838 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.116915 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.117360 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.117463 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.120307 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:10.120311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:10.120314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:10.120344 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:10.122796 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.123363 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.123368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.123370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.123490 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.123861 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.123867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.123870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.124191 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:10.124346 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:10.124717 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 13:19:10.124922 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:19:10.126051 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.109403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.0.97.62:38913)
0619 13:19:10.109543 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.109548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.109663 (+ 115us) server_negotiation.cc:407] Connection header received
0619 13:19:10.109777 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.109785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.109871 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.109962 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.111497 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.112547 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.113785 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.114389 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.114502 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.119340 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:10.119345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:10.119348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:10.119375 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:10.122673 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.123436 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.123441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.123445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.123605 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.123991 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.124000 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.124004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.124288 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:10.124539 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:10.125513 (+ 974us) server_negotiation.cc:299] Negotiation successful
0619 13:19:10.125736 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:19:10.126663 23377 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } has no permanent_uuid. Determining permanent_uuid...
I20260619 13:19:10.132100 23229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } }
I20260619 13:19:10.132408 23229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5716d3f17c91418c817a7a68ede34959, State: Initialized, Role: FOLLOWER
I20260619 13:19:10.132841 23229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } }
I20260619 13:19:10.133782 23384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } } }
I20260619 13:19:10.134297 23384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:19:10.134908 23229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 13:19:10 dist-test-slave-mkt3 krb5kdc[23149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 13:19:10 dist-test-slave-mkt3 krb5kdc[23149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 13:19:10.145503 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.127574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.0.97.61:41423)
0619 13:19:10.127716 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.127721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.127737 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:19:10.127849 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.127854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.127912 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.127982 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.129078 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.130133 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.131704 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.132330 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.132415 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.137240 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:10.137248 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:10.137254 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:10.137294 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:10.140841 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.141872 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.141879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.141883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.142040 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.142502 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.142507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.142511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.142869 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:10.143122 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:10.144903 (+ 1781us) server_negotiation.cc:299] Negotiation successful
0619 13:19:10.145149 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 13:19:10.147758 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.127350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45336)
0619 13:19:10.127738 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:10.127754 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:10.127870 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:10.128229 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:10.128233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:10.128672 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:10.128870 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:10.128882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.129825 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.129830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:10.130969 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:10.130982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.137958 (+ 6976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.137964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:10.138183 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.138635 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:10.138656 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:10.141193 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:10.145314 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:10.145320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:10.145323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:10.145781 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:10.146157 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:10.146162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:10.146164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:10.146288 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:10.146896 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:10.146898 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:10.147000 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 13:19:10.147098 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 13:19:10.147386 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296}
I20260619 13:19:10.147841 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.127350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45336 (local address 127.0.97.60:36535)
0619 13:19:10.127518 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.127523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.127803 (+ 280us) server_negotiation.cc:407] Connection header received
0619 13:19:10.127967 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.127972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.128025 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.128078 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.129030 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.129714 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.131099 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.131703 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.131830 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.141362 (+ 9532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:10.141367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:10.141371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:10.141404 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:10.144406 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.145919 (+ 1513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.145924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.145928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.146043 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.146378 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.146382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.146385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.146716 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:10.146845 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:10.147218 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 13:19:10.147521 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":60}
I20260619 13:19:10.150784 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 13:19:10.153172 23377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425: Bootstrap starting.
I20260619 13:19:10.153667 23204 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5716d3f17c91418c817a7a68ede34959"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53136
I20260619 13:19:10.154150 23204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 13:19:10.154390 23204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:19:10.156776 23204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } }
I20260619 13:19:10.156972 23377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425: Neither blocks nor log segments found. Creating new log.
I20260619 13:19:10.158324 23377 log.cc:824] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425: Log is configured to *not* fsync() on all Append() calls
I20260619 13:19:10.159832 23204 leader_election.cc:288] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [CANDIDATE]: Term 1 election: Requested vote from peers 9b704e0bbe2844ad8acd2a9989d219be (127.0.97.61:41423), c9c0b884473b4790a0351c6dd9e22425 (127.0.97.60:36535)
W20260619 13:19:10.165758 23395 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:19:10.166100 23395 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:19:10.169790 23377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425: No bootstrap required, opened a new log
I20260619 13:19:10.160948 23162 master_runner.cc:186] Error getting master registration for 127.0.97.61:41423: OK, instance_id { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" instance_seqno: 1781875149600956 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:19:10.174665 23162 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:19:10.180395 23377 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } }
I20260619 13:19:10.180823 23377 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9c0b884473b4790a0351c6dd9e22425, State: Initialized, Role: FOLLOWER
I20260619 13:19:10.181353 23377 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } }
I20260619 13:19:10.182718 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } } }
I20260619 13:19:10.183634 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:19:10.183744 23352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5716d3f17c91418c817a7a68ede34959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c9c0b884473b4790a0351c6dd9e22425"
I20260619 13:19:10.184286 23352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425 [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:19:10.187439 23352 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5716d3f17c91418c817a7a68ede34959 in term 1.
I20260619 13:19:10.188300 23180 leader_election.cc:302] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5716d3f17c91418c817a7a68ede34959, c9c0b884473b4790a0351c6dd9e22425; no voters:
I20260619 13:19:10.188802 23384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 13:19:10.183707 23377 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 13:19:10.198072 23384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [term 1 LEADER]: Becoming Leader. State: Replica: 5716d3f17c91418c817a7a68ede34959, State: Running, Role: LEADER
I20260619 13:19:10.198679 23384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } }
I20260619 13:19:10.204514 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.186627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53142 (local address 127.0.97.62:38913)
0619 13:19:10.186796 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.186802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.186899 (+ 97us) server_negotiation.cc:407] Connection header received
0619 13:19:10.187273 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.187279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.187360 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.187432 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.189710 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.190909 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.192851 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.193547 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.193651 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.197015 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:10.197022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:10.197028 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:10.197072 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:10.201043 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.201864 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.201870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.201874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.202054 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.202500 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.202505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.202509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.202925 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:10.203157 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:10.203786 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 13:19:10.204038 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 13:19:10.215871 23406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5716d3f17c91418c817a7a68ede34959. Latest consensus state: current_term: 1 leader_uuid: "5716d3f17c91418c817a7a68ede34959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } } }
I20260619 13:19:10.216322 23406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [sys.catalog]: This master's current role is: LEADER
I20260619 13:19:10.220607 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.187970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.0.97.60:36535)
0619 13:19:10.188129 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.188134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.205061 (+ 16927us) server_negotiation.cc:407] Connection header received
0619 13:19:10.205308 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.205314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.205417 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.205485 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.207278 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.208309 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.210136 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.210843 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.210935 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.214256 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:10.214263 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:10.214268 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:10.214310 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:10.217505 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.218312 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.218317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.218320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.218466 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.218904 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.218909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.218913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.219243 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:10.219505 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:10.220058 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 13:19:10.220275 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 13:19:10.229444 23410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 13:19:10.231721 23410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 13:19:10.234990 23307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.187965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.0.97.61:41423)
0619 13:19:10.188138 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.188144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.217567 (+ 29423us) server_negotiation.cc:407] Connection header received
0619 13:19:10.218263 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.218270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.218359 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.218429 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.222234 (+ 3805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.223269 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.225097 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.225710 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.225817 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.228637 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:10.228643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:10.228648 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:10.228685 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:10.231994 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.232755 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.232775 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.232779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.232936 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.233319 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.233325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.233328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.233691 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:10.233957 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:10.234288 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 13:19:10.234510 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 13:19:10.244293 23352 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425 [term 1 FOLLOWER]: Refusing update from remote peer 5716d3f17c91418c817a7a68ede34959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:19:10.245177 23384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 13:19:10.253188 23420 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 13:19:10.253401 23420 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 13:19:10.251961 23310 master_runner.cc:186] Error getting master registration for 127.0.97.61:41423: OK, instance_id { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" instance_seqno: 1781875149600956 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 13:19:10.263055 23310 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 13:19:10.266646 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5716d3f17c91418c817a7a68ede34959. Latest consensus state: current_term: 1 leader_uuid: "5716d3f17c91418c817a7a68ede34959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } } }
I20260619 13:19:10.266924 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:19:10.271196 23384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5716d3f17c91418c817a7a68ede34959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } } }
I20260619 13:19:10.271478 23384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [sys.catalog]: This master's current role is: LEADER
I20260619 13:19:10.271703 23422 mvcc.cc:205] Tried to move back new op lower bound from 7298560615387308032 to 7298560615219994624. Current Snapshot: MvccSnapshot[applied={T|T < 7298560615387308032}]
I20260619 13:19:10.274228 23410 catalog_manager.cc:1347] Generated new cluster ID: a6c310b34c3d413ea746eeb8289db088
I20260619 13:19:10.274323 23410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 13:19:10.282419 23402 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5716d3f17c91418c817a7a68ede34959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } } }
I20260619 13:19:10.282655 23402 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:19:10.308372 23410 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 13:19:10.311386 23410 catalog_manager.cc:1504] Loading token signing keys...
I20260619 13:19:10.321583 23303 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } attempt: 5
I20260619 13:19:10.334568 23410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959: Generated new TSK 0
Jun 19 13:19:10 dist-test-slave-mkt3 krb5kdc[23149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781875149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 13:19:10.336417 23410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 13:19:10.349108 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.322854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.0.97.60:36535)
0619 13:19:10.323005 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.323011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.323026 (+ 15us) server_negotiation.cc:407] Connection header received
0619 13:19:10.323101 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.323105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.323175 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.323263 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.324361 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.325496 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.330576 (+ 5080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.331426 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.331554 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.340115 (+ 8561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 13:19:10.340122 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 13:19:10.340127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 13:19:10.340171 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 13:19:10.344744 (+ 4573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.345521 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.345527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.345555 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.345816 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 13:19:10.346489 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 13:19:10.346495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 13:19:10.346499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 13:19:10.347070 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 13:19:10.347280 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:10.348089 (+ 809us) server_negotiation.cc:299] Negotiation successful
0619 13:19:10.348482 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 13:19:10.350412 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:10.322212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36535 (local address 127.0.0.1:45356)
0619 13:19:10.322539 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 13:19:10.322558 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 13:19:10.322671 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 13:19:10.323446 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 13:19:10.323451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 13:19:10.323994 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0619 13:19:10.324192 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:10.324204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.328739 (+ 4535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.328745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:10.330409 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 13:19:10.330425 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 13:19:10.332034 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 13:19:10.332040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 13:19:10.332240 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:10.333116 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 13:19:10.333141 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 13:19:10.337069 (+ 3928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 13:19:10.344897 (+ 7828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:10.344902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:10.344907 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:10.345419 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:10.346030 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 13:19:10.346036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 13:19:10.346061 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 13:19:10.346323 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 13:19:10.347417 (+ 1094us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 13:19:10.347422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 13:19:10.347672 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 13:19:10.349390 (+ 1718us) client_negotiation.cc:240] Negotiation successful
0619 13:19:10.349883 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213}
I20260619 13:19:10.357275 23303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be: Bootstrap starting.
I20260619 13:19:10.361207 23303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be: Neither blocks nor log segments found. Creating new log.
I20260619 13:19:10.362900 23303 log.cc:824] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be: Log is configured to *not* fsync() on all Append() calls
I20260619 13:19:10.382069 23303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be: No bootstrap required, opened a new log
I20260619 13:19:10.400641 23303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } }
I20260619 13:19:10.401374 23303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b704e0bbe2844ad8acd2a9989d219be, State: Initialized, Role: FOLLOWER
I20260619 13:19:10.402290 23303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } }
I20260619 13:19:10.414219 23277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be [term 0 FOLLOWER]: Advancing to term 1
I20260619 13:19:10.417641 23277 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be [term 1 FOLLOWER]: Refusing update from remote peer 5716d3f17c91418c817a7a68ede34959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 13:19:10.417925 23278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5716d3f17c91418c817a7a68ede34959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9b704e0bbe2844ad8acd2a9989d219be"
I20260619 13:19:10.421375 23278 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5716d3f17c91418c817a7a68ede34959 in term 1.
I20260619 13:19:10.423115 23384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 13:19:10.426501 23303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be [sys.catalog]: configured and running, proceeding with master startup.
W20260619 13:19:10.464332 23232 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:10.450830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53148 (local address 127.0.97.62:38913)
0619 13:19:10.451033 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.451040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.462379 (+ 11339us) server_negotiation.cc:407] Connection header received
0619 13:19:10.462662 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.462669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.462779 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.462854 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.464079 (+ 1225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53148: BlockingRecv error: recv got EOF from 127.0.0.1:53148 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 13:19:10.466737 23380 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:10.456967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45370 (local address 127.0.97.60:36535)
0619 13:19:10.457139 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.457145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.464047 (+ 6902us) server_negotiation.cc:407] Connection header received
0619 13:19:10.465224 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.465230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.465309 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.465376 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.466449 (+ 1073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45370: BlockingRecv error: recv got EOF from 127.0.0.1:45370 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 13:19:10.468637 23307 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:10.464493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.0.97.61:41423)
0619 13:19:10.464732 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.464738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.467153 (+ 2415us) server_negotiation.cc:407] Connection header received
0619 13:19:10.467386 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.467391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.467492 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.467561 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.468393 (+ 832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45574: BlockingRecv error: recv got EOF from 127.0.0.1:45574 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
I20260619 13:19:10.470140 23431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5716d3f17c91418c817a7a68ede34959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } } }
I20260619 13:19:10.470604 23431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:19:10.471101 23429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5716d3f17c91418c817a7a68ede34959. Latest consensus state: current_term: 1 leader_uuid: "5716d3f17c91418c817a7a68ede34959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } } }
I20260619 13:19:10.471453 23429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be [sys.catalog]: This master's current role is: FOLLOWER
I20260619 13:19:10.471614 23439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5716d3f17c91418c817a7a68ede34959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } } }
I20260619 13:19:10.471956 23439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be [sys.catalog]: This master's current role is: FOLLOWER
W20260619 13:19:10.515378 23307 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:10.502546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45590 (local address 127.0.97.61:41423)
0619 13:19:10.502728 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.502734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.513328 (+ 10594us) server_negotiation.cc:407] Connection header received
0619 13:19:10.513887 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.513894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.514037 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.514114 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.515119 (+ 1005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45590: BlockingRecv error: recv got EOF from 127.0.0.1:45590 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 13:19:10.522629 23458 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260619 13:19:10.542276 23307 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:10.540054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.0.97.61:41423)
0619 13:19:10.540236 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.540242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.540761 (+ 519us) server_negotiation.cc:407] Connection header received
0619 13:19:10.540943 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.540948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.541056 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.541167 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.542068 (+ 901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45604: BlockingRecv error: recv got EOF from 127.0.0.1:45604 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 13:19:10.559862 23307 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:10.551535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.0.97.61:41423)
0619 13:19:10.551691 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.551696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.558384 (+ 6688us) server_negotiation.cc:407] Connection header received
0619 13:19:10.558579 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.558586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.558686 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.558791 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.559609 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45618: BlockingRecv error: recv got EOF from 127.0.0.1:45618 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 13:19:10.569026 23307 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:10.566856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.0.97.61:41423)
0619 13:19:10.567071 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.567077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.567623 (+ 546us) server_negotiation.cc:407] Connection header received
0619 13:19:10.567773 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.567779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.567882 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.567988 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.568803 (+ 815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45624: BlockingRecv error: recv got EOF from 127.0.0.1:45624 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260619 13:19:10.582123 23307 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:10.580081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45634 (local address 127.0.97.61:41423)
0619 13:19:10.580252 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.580258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.580704 (+ 446us) server_negotiation.cc:407] Connection header received
0619 13:19:10.580903 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.580909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.581021 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.581130 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.581905 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45634: BlockingRecv error: recv got EOF from 127.0.0.1:45634 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 13:19:10.605525 23307 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:10.603256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.0.97.61:41423)
0619 13:19:10.603441 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.603447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.604008 (+ 561us) server_negotiation.cc:407] Connection header received
0619 13:19:10.604211 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.604216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.604317 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.604422 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.605232 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45646: BlockingRecv error: recv got EOF from 127.0.0.1:45646 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 13:19:10.644718 23307 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:10.642425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45648 (local address 127.0.97.61:41423)
0619 13:19:10.642624 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.642631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.643158 (+ 527us) server_negotiation.cc:407] Connection header received
0619 13:19:10.643412 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.643418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.643509 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.643609 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.644468 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45648: BlockingRecv error: recv got EOF from 127.0.0.1:45648 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 13:19:10.714184 23307 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:10.712331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.0.97.61:41423)
0619 13:19:10.712546 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.712553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.712898 (+ 345us) server_negotiation.cc:407] Connection header received
0619 13:19:10.713070 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.713076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.713169 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.713278 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.713974 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45654: BlockingRecv error: recv got EOF from 127.0.0.1:45654 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 13:19:10.847419 23307 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:10.845728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45666 (local address 127.0.97.61:41423)
0619 13:19:10.845981 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:10.845987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:10.846190 (+ 203us) server_negotiation.cc:407] Connection header received
0619 13:19:10.846387 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:10.846392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:10.846461 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:10.846546 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:10.847236 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45666: BlockingRecv error: recv got EOF from 127.0.0.1:45666 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 13:19:11.027923 23384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5716d3f17c91418c817a7a68ede34959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5716d3f17c91418c817a7a68ede34959" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38913 } } peers { permanent_uuid: "9b704e0bbe2844ad8acd2a9989d219be" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41423 } } peers { permanent_uuid: "c9c0b884473b4790a0351c6dd9e22425" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36535 } } }
I20260619 13:19:11.028270 23384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5716d3f17c91418c817a7a68ede34959 [sys.catalog]: This master's current role is: LEADER
W20260619 13:19:11.108206 23307 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 13:19:11.106689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.0.97.61:41423)
0619 13:19:11.106852 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:11.106857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:11.107097 (+ 240us) server_negotiation.cc:407] Connection header received
0619 13:19:11.107253 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:11.107257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:11.107330 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:11.107421 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 13:19:11.108037 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45672: BlockingRecv error: recv got EOF from 127.0.0.1:45672 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 13:19:11.255399 23420 catalog_manager.cc:1259] Loaded cluster ID: a6c310b34c3d413ea746eeb8289db088
I20260619 13:19:11.255507 23420 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425: loading cluster ID for follower catalog manager: success
I20260619 13:19:11.257965 23420 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425: acquiring CA information for follower catalog manager: success
I20260619 13:19:11.259652 23420 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c9c0b884473b4790a0351c6dd9e22425: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:19:11.526964 23458 catalog_manager.cc:1259] Loaded cluster ID: a6c310b34c3d413ea746eeb8289db088
I20260619 13:19:11.527081 23458 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be: loading cluster ID for follower catalog manager: success
I20260619 13:19:11.529693 23458 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be: acquiring CA information for follower catalog manager: success
I20260619 13:19:11.531504 23458 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b704e0bbe2844ad8acd2a9989d219be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 13:19:11.633330 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:11.626727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.0.97.61:41423)
0619 13:19:11.627524 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:11.627529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:11.627545 (+ 16us) server_negotiation.cc:407] Connection header received
0619 13:19:11.627614 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:11.627618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:11.627764 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:11.627866 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:11.628409 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:11.629196 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:11.631086 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:11.631500 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:11.631608 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:11.632170 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:11.632737 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:11.632844 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:11.632971 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 13:19:11.633038 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":539,"threads_started":1}
I20260619 13:19:11.648073 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:11.639434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45384 (local address 127.0.97.60:36535)
0619 13:19:11.642170 (+ 2736us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:11.642176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:11.642195 (+ 19us) server_negotiation.cc:407] Connection header received
0619 13:19:11.642300 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:11.642305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:11.642515 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:11.642683 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:11.643133 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:11.644227 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:11.645508 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:11.646138 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:11.646261 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:11.646758 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:11.647435 (+ 677us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:11.647585 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:11.647683 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 13:19:11.647764 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2617,"thread_start_us":2521,"threads_started":1}
I20260619 13:19:11.668133 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:11.653987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.0.97.62:38913)
0619 13:19:11.661458 (+ 7471us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:11.661462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:11.661486 (+ 24us) server_negotiation.cc:407] Connection header received
0619 13:19:11.661576 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:11.661580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:11.661720 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:11.661901 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:11.662494 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:11.663526 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:11.665523 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:11.666147 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:11.666237 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:11.666894 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:11.667338 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:11.667430 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:11.667642 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 13:19:11.667741 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7362,"thread_start_us":7272,"threads_started":1}
I20260619 13:19:11.668540 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:11.654251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.0.97.60:36535)
0619 13:19:11.654407 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:11.654413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:11.662578 (+ 8165us) server_negotiation.cc:407] Connection header received
0619 13:19:11.662805 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:11.662810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:11.662904 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:11.662963 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:11.663650 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:11.664669 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:11.666656 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:11.667186 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:11.667272 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:11.667835 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:11.667949 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:11.668007 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:11.668212 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 13:19:11.668286 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 13:19:11.674129 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 23162
I20260619 13:19:11.674912 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 13:19:11.662696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.0.97.61:41423)
0619 13:19:11.662856 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 13:19:11.662862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 13:19:11.668838 (+ 5976us) server_negotiation.cc:407] Connection header received
0619 13:19:11.669135 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 13:19:11.669141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 13:19:11.669227 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 13:19:11.669285 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 13:19:11.669780 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:11.670708 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:11.672478 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 13:19:11.673042 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 13:19:11.673166 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 13:19:11.673605 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 13:19:11.673732 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 13:19:11.673807 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 13:19:11.674560 (+ 753us) server_negotiation.cc:299] Negotiation successful
0619 13:19:11.674642 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 13:19:11.674947 23224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:19:11.792560 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 23162
I20260619 13:19:11.809911 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 23236
I20260619 13:19:11.811239 23298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:19:11.924556 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 23236
I20260619 13:19:11.940376 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu pid 23310
I20260619 13:19:11.941488 23372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 13:19:12.046312 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwhPxxl/build/asan/bin/kudu with pid 23310
2026-06-19T13:19:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4549 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8855 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (344711 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST